JS21

Rural public water system in TROUBLE

17 posts in this topic

Posted (edited)

Hello all, I am new here. I have some basic experience with industrial automation and programmable devices like PLCs and vehicle engine computers, as well as C#. Now, I ALSO have a big stack of huge problems regarding a neglected/abandoned Public Water System (EPA regulated) that a small rural community (approx 200 souls) depends on.  After discovering that our water system had been abandoned and all its money embezzled by a now-dead crook, I took charge of the situation and got the system powered back up and semi-functional again. However, we have a high-priority issue brewing... the water treatment plant is operated via a Mitsubishi Alpha2 PLC (14 channel model, with Analog output module) and two Variable Freq. Drive units: An Eaton SVX9000 sized for a 30HP motor, and a Mitsubishi F800 sized for 7.5HP motors.

From 2006 to Dec 2021, the active program loaded on the PLC has been the ONLY copy of the program in existence! When I discovered the PLC and realized there was no backup of the program and no Serial cable to retrieve a backup, I immediately researched the AL2 unit and found a Serial cable on eBay and also purchased a spare AL2 and Analog module. I have successfully obtained the AL-PCS/WIN 2.70 software and extracted a copy of the program. At this point, the emergency has been dealt with. However...

The PLC program is incomplete and partially non-functional. It is not behaving as expected according to the limited Distribution System drawings I have available. I want to debug and fix the program, but now I have run into a SERIOUS problem trying to make sense of the extracted program.  Because I don't have the original file that was used to render the final program and write it to the hardware, the "map" of where all the functions and objects should be graphically laid out within the editor has been lost. When I open the raw file in the Alpha2 editor software for the first time, the editor alerts me that a file is missing and it must reconstruct the visual representation of the layout from scratch. 

When it imports the raw file, the entire layout is crammed into a ~640 pixel wide column that takes 5 or 6 pages to scroll through!  It is impossible to reverse engineer the PLC program when it looks like this! To make matters worse, there is no "group objects" function that I can find, and when I expand the width of the layout, I can only drag ONE item at a time to a new position... if I try to multi-select two or more objects and move them, the GUI editor freaks out and scatters the selected icons all over the layout.  I have already tried once, spending about 20 hours, trying to re-organize everything, but it is extremely labor intensive and I have barely made any real progress!

lWqE0nS.jpg

PLEASE tell me there is a way to repair this layout without spending another 40 hours dragging every icon around 100 times. It feels like an exercise in futility at this point, especially since the program is less than 5000 bytes -- but I have no other alternative...this software runs a critical piece of public infrastructure and I NEED to be able to maintain it.  Any help will be greatly appreciated by me, and my little community who can't drink or shower without this rat's nest of a program:

ASde5nb.jpg

 

What a nightmare!  This is as far as I got, and I am about to abort the attempt and start over from scratch, because now the program seems to have new malfunctions.
 

Edited by JS21

Share this post


Link to post
Share on other sites

first of all congrats on taking charge of the situation and making thing better.

i only used Alphas couple of times and that was years ago. one of the big problems i recall is exactly what you are describing - code representation. code does not retain state and that alone is a death sentence to anything more than a most trivial exercise since it makes code maintenance practically unusable.

this is by far the biggest complaint i have with FBD language on any platform, but in case of a small controller (smart relay) this is particularly painful as one cannot organize code in separate files or pages. i would have hoped that over time more control over this is added but i guess that was not the case. 

i do not want to use programming language unless code representation is rigid. that means sticking with ladder and structured text. i really do not want to waste my time dragging boxes and lousy rubber band connections.... 

since there is only handful of I/O, changing platform should be easy. i would just replace it with something newer, more flexible and much easier to use.

for example Click PLC from AD is robust, very low cost, modular and software is free. using ClickPlus you also get Ethernet option so this can be tied to network, etc.

good luck...

 

Share this post


Link to post
Share on other sites

If you go to Automation Direct, I highly recommend skipping directly to the Productivity series.  It has superior support for external devices (like drives) due to support for EtherNet/IP.

Share this post


Link to post
Share on other sites

Posted (edited)

I am not opposed to switching platforms, but right now we have NO budget for it, and have to stick with what we've got.  If the PLC dies, then 200 people can't take a shower and their homes become worthless. We can't even shut the system down for a few hours without requiring an expensive EPA-approved startup procedure that we don't have any documentation on yet, and if we don't follow the startup procedure then we risk bacterial contamination and more EPA violations.

Even though the program is not very complicated with only 4 inputs and 6 outputs, the logic between the Ins and Outs is entirely undocumented, so the problem isn't as much an outdated platform as it is a lack of understanding how the current program works. It is driving 2 well pumps, 2 tank pumps, and 1 chlorine injector, but that is literally all we know at this point. None of the wires connected to the PLC are labeled, none of the logic inside the PLC is known to any living person. It is supposed to have "fire suppression mode" and be able to respond to other unusual/emergency situations, but that is all just hearsay and shreds of information I was able to glean from neighbors and old handwritten notes from the deceased system operator.  The current program must be understood before it can be recreated on a different platform, and the switch from one platform to another must be done as quickly as possible because every minute of downtime costs us money that we simply don't have.

It looks like there is no way to recover the graphical layout of our Alpha2 program besides manually dragging one icon around at a time until it looks and works right again. I was hoping there would be a newer version of the software, or a 3rd party / open-source alternative that can edit the files in a more efficient way.  I'll keep plugging away at it.

Edited by JS21

Share this post


Link to post
Share on other sites
1 hour ago, JS21 said:

costs us money that we simply don't have.

Start making plans.  In 35 years in this business, I've seen only one case where PLC memory was corrupt, uncaught, and affected the operation.  All other cases of "misbehavior" was either a long-standing bug, or hardware failure.

I think it highly likely you are experiencing hardware failure.  And if you don't collect what you need from the homeowners to fix it, you will be collecting from the homeowners to pay lawyers, pay more fines, more lawyers, and then pay outsiders to fix it (by court order).

Share this post


Link to post
Share on other sites

agreed....

btw. access to water is vital and will easily gain sympathy. a good word and plea for help may go long way. cost of replacement hardware that would make job easier is really not high. single donation could take care of it. maybe start page at GoFundMe and cross fingers...? or consider asking AD for a possible hardware donation. 

 

Share this post


Link to post
Share on other sites

Heck, if you are anywhere near Atlanta, I'm willing to offer a complimentary consult.  (I'll admit to AD and AB biases, fwiw.)

2 people like this

Share this post


Link to post
Share on other sites

:-2

Share this post


Link to post
Share on other sites

Posted (edited)

4 hours ago, panic mode said:

agreed....

btw. access to water is vital and will easily gain sympathy. a good word and plea for help may go long way. cost of replacement hardware that would make job easier is really not high. single donation could take care of it. maybe start page at GoFundMe and cross fingers...? or consider asking AD for a possible hardware donation. 

 

Not that simple. We had plenty of money, but it suddenly vanished. There are legal issues preventing us from accepting funding beyond 10% of our currently-zero budget. It is more than I am able to get into here, but suffice it to say, the people who funded the district already made lifetime investments and can't afford to double-down.
 

3 hours ago, pturmel said:

Heck, if you are anywhere near Atlanta, I'm willing to offer a complimentary consult.  (I'll admit to AD and AB biases, fwiw.)

I appreciate the offer, but we're in the middle of nowhere in the grasslands of the USA. We're relying on free help at this point, and have funding for approximately 13 more months of operation. Doubling water rates would buy us 13 more months max.

All I can do right now is make sure we don't have any system downtime by obtaining spare hardware and copying our one-of-a-kind program to a backup PLC, and then identify & label all the wires connected to the PLC. Second priority is to reverse engineer the program in order to bring an inactive well/pump back online. The well has been offline for months and we're running on a single well, using a pump of unknown age/condition. We are supposed to be alternating between two wells and two tanks, but the lead/lag functions are not doing anything when enabled.
More important than the lead/lag function is the chlorine injection function. It is supposedly driving a proportional chlorine injector machine via Analog outputs, but it appears to me that the analog outs are merely being used as relay triggers. All the outputs ever do is switch from 0 to 5 volts now and then. That is not how proportional chlorine injection is supposed to work, but we depend on it for continuous chlorine residual maintenance which must be monitored and reported to the State every 24-48 hours.

Edited by JS21

Share this post


Link to post
Share on other sites
Quote

We had plenty of money, but it suddenly vanished.

sounds like you have no money now.

Quote

There are legal issues preventing us from accepting funding beyond 10% of our currently-zero budget.

sounds like you may not be able to accept funds. not sure that includes hardware donations. (just wag)

Quote

 All the outputs ever do is switch from 0 to 5 volts now and then.

maybe using hysteresis or PWM to achieve the same? need to check the logic...

 

Share this post


Link to post
Share on other sites
19 hours ago, JS21 said:

Even though the program is not very complicated with only 4 inputs and 6 outputs, the logic between the Ins and Outs is entirely undocumented,

 

With that low number of I/O  I would bet its mostly an order of operations & timer based system.

It might be a good idea to try to create a map of the functions you KNOW it does and build from there.  A scope or meter on the outputs to get a timing and order. if there is a HMI look for any settings / parameters.

Share this post


Link to post
Share on other sites

I finally got the FBD laid out enough to start labeling I/Os, default parameter values and some of the logic. Painstaking work, because I can only move one icon at a time. Moving two scatters them randomly around the layout and I have to revert to the previously saved revision.

 

AGcVoM9.jpg

Share this post


Link to post
Share on other sites

@JS21 
Hi,
Perhaps I could help your community to solve problems for a modest fee that doesn't bust your budget.

Share this post


Link to post
Share on other sites
On 1/9/2022 at 6:14 AM, Inntele said:

@JS21 
Hi,
Perhaps I could help your community to solve problems for a modest fee that doesn't bust your budget.

 

Tell me more!  Do you think you can repair this program, or are you thinking of starting from scratch? It seems impossible to start a new program without having physical access to the system to observe and tune the logic.

Share this post


Link to post
Share on other sites
1 hour ago, JS21 said:

 

Tell me more!  Do you think you can repair this program, or are you thinking of starting from scratch? It seems impossible to start a new program without having physical access to the system to observe and tune the logic.

To begin with, it would be right to unravel the tangle by sorting the chains of functions into the corresponding individual functional blocks, which should, firstly, make it possible to figure out how everything was implemented, and, secondly, it would simplify making changes to the program. Theoretically the program can be written from scratch, however I haven't worked with AL2 for a long time so forgot details how to work with buttons and how to organize correctly a storing of setpoint values and their changes by abovementioned buttons.
Three components allows to carry out adjustment work remotely:
- a person at the facility, which can to connect/disconnect cables, to control the process, to make and send photos and videos;
- skype;
- team weaver.

Share this post


Link to post
Share on other sites
On 4/14/2022 at 7:56 AM, Inntele said:

To begin with, it would be right to unravel the tangle by sorting the chains of functions into the corresponding individual functional blocks, which should, firstly, make it possible to figure out how everything was implemented, and, secondly, it would simplify making changes to the program. Theoretically the program can be written from scratch, however I haven't worked with AL2 for a long time so forgot details how to work with buttons and how to organize correctly a storing of setpoint values and their changes by abovementioned buttons.
Three components allows to carry out adjustment work remotely:
- a person at the facility, which can to connect/disconnect cables, to control the process, to make and send photos and videos;
- skype;
- team weaver.

I can provide pictures, videos and detailed information about the physical setup, can also be on Skype/TeamViewer/etc. I have spent a significant amount of time trying to unravel, and eventually had to stop working on it and focus on my day-job again. How long do you think it would take for you to unravel, and at what price? De-obfuscation of the program so that all the functions can be reverse-engineered is definitely the first step.

Share this post


Link to post
Share on other sites
13 hours ago, JS21 said:

I can provide pictures, videos and detailed information about the physical setup, can also be on Skype/TeamViewer/etc. I have spent a significant amount of time trying to unravel, and eventually had to stop working on it and focus on my day-job again. How long do you think it would take for you to unravel, and at what price? De-obfuscation of the program so that all the functions can be reverse-engineered is definitely the first step.

I'm afraid that at current situation just I couldn't help. Three and a half months have passed since the publication of my proposal for your appeal. Now Russia is under sanctions for invading to Ukraine. Russian banks are also under sanctions. As far as I know, the sixth sanctions package, which is to be announced next week, provides for the exclusion of the remaining two banks from the SWIFT settlement system. Systems of transcontinental money transfer have already left Russia. So you just won't be able to pay me this work.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now