Sign in to follow this  
Followers 0
Crossbow

Amazing lack of intelligence in automation industry...

12 posts in this topic

Both at work and here in the forums I see many new posts looking for help on platforms which are 20 years obsolete and complaints about ancient software not working with new operating systems. God I hate these situations, but it clearly points to a lack of understanding on the part of companies with automated equipment. It's sad that more companies do not understand that PLCs are computers, and they replace desktop and laptop computers every 5 years but hope and pray to support 20 year old PLCs until the next millenium. I hate working in that environment. A PLC is a computer, the same components are used inside, they operate in a worse environment, and most companies are too penny-pinching to replace them on a regular basis. I bet not one of them drives a 20 year old car to work every day, Or use a cell phone more than 2 years old. But they let the operation of their critical machines rely on outdated and obsolete technology and then complain when one goes down and they cannot support it. If you read many manufacturers provide recommendations in their manuals even to change them out after 10 years. I've been doing this for 16 years, and I have worked many times on PLCs which were obsolete before I was out of college. Businesses who use automated equipment should make a plan for that equipment as soon as they buy it. That plan should include preventive maintenance, lubrication, battery replacement, and whatever else has to be done to keep it running. It should also include regular backups of the controller programs, and a plan to replace the controllers when the manufacturers announce the products are becoming obsolete. Electronics change frequently these days, and components from even 10 years ago are becoming very hard to find. Look at flash memory. Compact Flash was the way to go 5-6 years ago, now almost nobody still makes it, and the choices for sizes have become very limited. This process happens with all electronics, industrial or commercial. It's called progress people! See this as a rant if you will, but to me this is just common sense. But we all know common sense is not so common anymore.
4 people like this

Share this post


Link to post
Share on other sites
I know one of the issues some owners must deal with is approvals. Either the hardware/software was approved out of the box or a system was designed and submitted for approval. Once it is approved it can be a hard road to get approval again, not to mention the cost. I worked, for years, for a company that manufactured out of the box embedded approved systems, CSA, TUV, UL, NFPA, etc. and talking about changes to the system after spending $500,000 for all the approvals can get you many evil eyes. Some changes can be approved with a paper submittal and some require a complete round of testing and that can add the cost. For systems that are not out of the box, but are 'application' approved, the site system is approved, the owner normally does not have in house talent to get any changes approved. Which then leads to hiring a company/integrator/manufacture to do the work and get it approved. The cost of changing the hardware is the least expensive area. As hardware/software for embedded systems gets more complicated, one round of a board (hardware) change, during development, can cost $50,000. Now do it ten years after release (because some IC is out of production, etc.) and the cost can really climb. Assuming the team that created the first release is still around, if not, that can really add cost as the learning curve can become extended. So, the moral, for me is as the OP stated, be aware that things will break/wear out and need replacement. If you do not have spares you could be in a real pickle. Include a large number of spares in the original work/contract/project and when you burn though to 50% remaining, start looking for solutions, more spares or replacement. My2c.

Share this post


Link to post
Share on other sites
It's the same everywhere, exept we possibly get stuck in the middle of the ocean if one of these old PLC's knock out. (No Fish = No Paycheck) Stearing Gear controled by 25 year old IDEC. (we have no program backup, software or spare parts) Electrical Distribution Switch Gear and Protection controled by 25 year old Siemens S5. (we have no program backup, software, or spare parts) Our main Factory is running on 20 year old PLC5 (but at least I have the program and software for this one, and some 20 year old spare cards) Some newer things that have been added are runnning on Micrologix and Control Logix, but try and get spare parts to have at hand on board. (not a chance usually) No one in corperate will even consider replacing them or even have someone come down who has the software for these and pull the program, so we can start converting the program over to A.B. and have a back up plan ready for the day it does go poof. Keep fighting the good fight, but dont let it give you High Blood Pressure. Sometimes all we can do is: ALL THAT WE CAN DO. BCS Edited by Bering C Sparky

Share this post


Link to post
Share on other sites
Unfortunately PLC replacement programs seem to be controlled by two old addages, "Out of sight, out of mind" and "If it ain't broke don't fix it". Most PLCs are extremely reliable so they tend to last much longer than the parts supply chain does. Very few customers are willing to shelf stock spare parts. All of this leads to the problems that you have described and warnings to the customer of impending doom seldom do any good. I have one system rignt now (Omron CQM1) which was obsolete when the machine was installed and is now difficult (but not impossible) to find parts for. The customer won't allow a retrofit because "that takes too long and we're behind on production". It's only a matter of time until this machine suffers a permanent failure. I have another customer replacing a 1970's control system for a waste treatment plant with all new components. They decided that the recommended spare parts list was not necessary because all the parts are currently available. Of course by the time they have a failure the parts won't be available anymore! Edited by Mendon Systems

Share this post


Link to post
Share on other sites
I remember an old system I worked on that was a fore-runner to PLC's. It had a mechanical stepper and diode plug boards to control steps and transitions. Gives me shivers to remember the hell we went through to keep that thing running. No spare parts to be found, all long gone and obsolete. We lobbied continuously to gut the panel and replace it all with a PLC and were continually knocked back because "We can't stop production." That is until it finally went belly up and we couldn't fix it any more. No choice then but to upgrade it. When the project was finished, and got it all back up and running, downtime went to practically zero, cycle times were substantially reduced and production went through the roof. About a week after this, management said "Do the others." There were three more to do. Previously, all four had to keep running flat out to barely keep up with requirements. After the upgrades, one machine could be held back for preventative maintenance while the others were in production. Edited by Andy_P

Share this post


Link to post
Share on other sites
I know about approvals and such as mentioned by Mark-... I dealt with a government project where the part number on the Siemens Logo changed due to version upgrade. They halted the panel builds and demanded the new control panel be recertified even though Siemens stated it was 100% drop in replacement for the old one. What a bunch of BS for part number change -0BA3 to -BA4. But the real focus of my original post was that 'corporate' will say we can't spend money and can't take a machine offline. Sooner or later, the machine will fail and take itself offline. And then 'I told you so' just isn't going to cut it. And then management is blaming maintenance for not keeping the machine up. I strongly recommend to all students I have trained to make sure they have a plan. I've been teaching customers on factory automation for years. And I always tell them to remember that what they are using today will not be available in 10 years, so keep that time frame in mind.

Share this post


Link to post
Share on other sites
I'm constantly amazed at the OEMs that no longer have a controls engineer. They had one guy when the company was founded that wrote a decent PLC program and they've had interns and service guys hacking and patching at it ever since. They can't ever change the machine or the controls platform and continue to make exactly the same machine with the same controls for decades until they are put out of business by a more innovative competitor or they can't buy the PLC anymore. Also shocking to me is that a lot of companies don't use proper version control. I mean, it's free to slap a Git repository on BitBucket and track your changes and every non-PLC programmer uses it for everything, but in the PLC world, no body knows the who, when, why, or where to find the last change in the PLC code. And then there's ladder logic... Let's just use a cumbersome system of hieroglyphics that are different from platform to platform as the industry standard. Why? Because it's easy for the shop guys to understand? Maybe if they're illiterate.

Share this post


Link to post
Share on other sites
Most modern controllers offer more programming languages than just ladder, and each has its strengths and weaknesses. I am seeing much more structured text programming, and less ladder in newer programs. It all depends on the end user.

Share this post


Link to post
Share on other sites
Thank you for this post! It's like counseling! There is staggering incompetence in this industry, it's good for business (we do a lot of rehab projects), but from a humanity perspective it's hard to stomach at times. I just got back to PLC forums yesterday for fun, and so many of the questions leave me scratching my head with how the situations described ever were allowed to evolve. Mythical Man Month (book) provided some insights, unlike concrete or pipes, software is abstract and hard to visualize, thus people get themselves deep in trouble before they realize it. I've brainstormed your exact complaint numerous times through my career and I also think it's related to marketing. The vendors/manufactures commonly advertise that their platforms make it easy for any stakeholder to use. When in fact this is a craft that requires discipline and sound engineering methodologies. Systems integrators are often just as bad. Some clients quickly lose site of long term vision also, allowing non-standard equipment/methods to creep in for short term reasons. Wouldn't it be cool if there was a taxi-like meter that showed a client how much their decision would cost them in 5yrs, 10yrs, 20yrs? "Just change that real quick, no big deal run the wire over there, and document it later, we have to get going!" In 10 years when you need a platform upgrade, it's going to cost $12k for your integrator to unwind that portion. Glad to know there are like minds out there. Edited by SoCalIntegrator
1 person likes this

Share this post


Link to post
Share on other sites
Version control sucks if all you get are binary files and a change comment. None of the PLC software that I use on a regular basis makes it easy enough to get a plain text representation that would be suitable for a VCS. I'm sure that's somewhat related to everything being ladder logic.

Share this post


Link to post
Share on other sites
My brother-in-law is an electrician for a utility. He's learned to repair the boards in their PLC's because they won't buy new ones. I think management sees more risk than benefit. The cascading cost of replacing the PLC causing a need to replace other parts, the unknown of what will and won't be compatible, bugs due to the conversion. We are going through that right now. Lots of software bugs cropping up simply due to the decreased scan times of the new PLC's. Of course we no longer allow scan based logic, but it can't be helped when converting old code unless you want to spend to time to re-write. Also, replacing a PC isn't really a fair comparison. PLC's cost much more in hardware, not to mention labor. You're looking at a couple thousand vs tens of thousands up to hundreds for PLC replacement. Of course, the PLC is controlling something that generates a lot more profit than the PC.

Share this post


Link to post
Share on other sites
But if the PLC goes down, no amount of money saved by not upgrading it makes up for the lost production. And every PLC out there offers settings for a fixed cycle time. So slow the new processor down to match the old one.
1 person likes this

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0