Redundant PLC Systems

[BobLfoot 302]

With more and more vendors offering redundant control solutions I thought a post discussing experiences and pros and cons was worthwhile. Hope some agree. Allen Bradley ControlLogix and ControlNet redundant solutions. I've used these both in version 8.5 with software redundancy and version 13 with SRM Module hardware redundancy. The system designer was trying to use redundancy in a real time warehousing application and as such this was a less than desirable application. To avoid errors we were forced to operate at 4 to 6 inch resolution on our transportation belts. We also had to watch scan time like a hawk and keep it in the 15 to 25 ms range. If an online edit forced scan time above 35 ms we had also sorts of prolems. Edited 3 Apr 2006 by BobLfoot

[Ken Moore 23]

As a long time DCS guy, IMHO PLC's have a long way to go before they approach the level of reliability that a DCS can have. Don't get me wrong, I believe there is a place for both. In the last few years a lot of the functions of a DCS are now available in the top shelp PLC's, but they still have some catching up to do. We have numerous SLC's, (CLX's coming this summer), MircoLogix's and PLC-5's also, it's just a matter of selecting the right platform for the particular application. All my DCS experience is with Foxboro I/A, and when I type DCS, I mean Foxboro. Anyway, in the DCS redundancy is handled by the system design, it does not require any special programming, you just configure the system as redundant and everything automatically works (magic!). The switching is automatic, the monitoring is automatic. The down side is price. But if you really need the redundancy, price is usually not the deciding factor. I have had the Rockwell Process guy in trying to push the redundant CLX line, but when you dig into the requirements, it's not as easy as they would have you believe. So until they get it better, I'll stick with the DCS for critical applications. I know I'm a little off topic, but thought you PLC only guys might like to hear what is possible on other platforms.

[TimWilborne 108]

I have never done redundant PLCs and I am still having a hard time understanding at which point you should have redundancy. Yes, critical systems need redundancy. But at what time is redundancy just an overkilled convenience or almost paranoia

[DanW 41]

I just did a small redundancy project with a Honeywell hybrid controller. As a hybrid, it is designed for process, not sequence, so it isn't designed for speed and throughput like a conventional PLC. It programs in function block. There were assurances from Honeywell that redundancy would not affect processing time. Loops are processed on a 500mS basis, so that isn't surprising. The project was about 200 I/O points, fit into 2 racks. The redundancy included redundant controllers with a switchover module, redundant power supplies, but no redundant I/O. It could have had redundant ethernet to the HMI, but the customer was only concerned about actual processor redundancy. The redundancy is implemented in the hardware. Like Ken Moore mentioned about Foxboro IA, for this box there was no special programming, it just runs redundant when you turn it on. There is a system function block that has a taggable DO that indicates whether the controller has switched for failure reasons. The value of the processed material is in an given batch is in 7 figures, so an addtional $12K for redundancy was no big deal. Dan

[BobLfoot 302]

So let me get this correct Dan. A batch costs 7 figures and they spent 12k on CPUs and Power Supplies, but will let a $30 sensor cost them the whole batch. Redundant IO would ahve made more sense than the CPUs to me.

[TimWilborne 108]

This is kind of what I was saying about the overkill. How often does a processor go bad or a network segment. A sensor is much more likely to be a culprit. Or even more a bearing or something mechanical. And while I can't say this for sure, but judging by the post you see on MrPLC I only hear about processor redundancy. Exception to my argument - Safety and critical systems can't have enough redundancy

[jstolaruk 9]

The only place I've worked on PLCs with a hot backup was in automotive paint booths and at the time they were PLC3s and had a whole lot of I/O with the robots. I suspect it was for the environment (explosion proof equipment everywhere), and the additional cost was trivial compared to having the line down. Other than that I don't recall seeing true redundant systems in the auto mfg biz besides safety PLCs.

[Ken Moore 23]

For Really critical applications, you have redundant everything, including I/O modules, and we use 3 sensors, with 2 out of 3 voting. So the loss of one instrument will not cost you the batch.

[DanW 41]

I didn't mention, it is NOT a safety system. It is a primarily a cascaded temperature control system on a batch that takes weeks to process with some batching functions. I agree, I/O is the most likely failure point, although power supplies run a close 2nd. I heard that the division that makes this hybrid controller can't have redundant I/O for marketing reasons because redundant I/O would compete with Honeywell's DCS systems. So the work around includes a pair of $5K specialty temperature transmitters (that they were already using) that analyzes the junction for drift, in addition to producing a temperature output. There are additional multiple dual element RTDs programmed for fall back if the specialty unit signals drift. The RTD elements are checked against one another for deviation and points are checked one another for deviation. They view the level of redundancy that they got as an insurance policy.

[Money4Nothing 0]

I design and program several redundant control systems. The Controller, I/O and Media are all redundant on one, and on another only the Controller is redundant. Its as simple as thinking through the scenarios of what could happen if a part of your control system failed, and whether you are willing to live with the consequences. Usually I have used the Allen Bradley Control Logix SRM module, which is great (if expensive). Sometimes I manually create a redundant system by using duplicate components and interlocking them with wiring and programming. $

[Big Country 3]

I have worked on some projects that require 3 times redundancy, it was for a handi-cap accesible elevator that are installed in old historic buildings. But that is the only time I have ever seen it specified on a project.

[ssommers 20]

I haven't done a redundant system myself, but I would have used for this voting method on a job where the customer was destroying old chemical munitions in an oven. I wasn't the controls engineer - I was the project manager's documentation control specialist (aka highly-qualified paper pusher) - so I didn't get to voice my opinion until the drawings were ready to submit to the customer & then it was too late. The control cabinets were 300 ft away from the oven and it could take a person a couple hours to suit up, go into the "dirty room" & fix that little sensor problem. Personnel safety definitely trumped the cost of redundancy, but they still only used redundant processors & not redundant sensors as well. I thought that was silly, but it was a government contract & that was the minimum requirement... Which brings up another question... When you have 2 sensors for the same point, how do you know which one failed? I understand how it works for thermocouples (they tend to go to the end of the spectrum), but how could you tell that one prox had failed vs the other? Thanks, Sue "We can lick gravity, but sometimes the paperwork is overwhelming." -- Werner von Braun Edited 20 Apr 2006 by ssommers

[Ken Moore 23]

In the case you mentioned, I would have used three sensors. Then you can utilize 2oo3 voting. With only two sensors, you would have to use 2oo2. So anytime the two disagreed, you would have to shutdown.

[catserveng 0]

We do mostly switchgear lineups for power generation systems, about 10% of the systems we supply require "hot backup". In all but one case, we have never had a problem with a system that required the hot back up to take over, seems if we do things right up front, the processor does pretty darn good, and whatever will kill a processor in our business, like a close lightening strike, usually kills a lot of the electronics. I find redundant systems very hard to commission and service in the field, and usually cause more problems in our business than they cure. My experience is primarily with AB SLC and PLC5 systems, and two systems using GE 90-30's. We are currently installing a CLX system that will startup in July, we'll see how that goes.