Disqualified Redundancy Partner IO Issue?

[GuyMiller 1]

I am investigating a failure that occurred...  We have a L62 Processor with RIO and Redundancy, it was reported that a system "failed to stop" this system is started and stopped via outputs channels on an OW16I card on one of the RIO racks. Has anyone had experience an issue with IO not responding as it should if the redundant partner is disqualified?

Also, there was a know network issue and the main Ethernet card (Slot 1) did have a communication issue (not the same Ethernet card for the RIO (Slot 3).

Edited 22 Jan 2020 by GuyMiller
More Information

[Ken Roach 166]

Obviously that's a light load of details to work on, but you've got what you've got.

First, please verify that when you say "RIO" you mean I/O on EtherNet/IP, through a 1756-EN2T module.     The term "RIO" is a bit of legacy vernacular in the A-B world, referring to the old "blue hose" Universal Remote I/O network that the PLC-2/3/5 used for decades.

Do you think that the comms problem on the Slot 1 Ethernet module was the reason that the Secondary was disqualified ?

Did the disqualification and the failure to function as expected occur at the same time, or close together in time ?  

Do the loads connected to the 1756-OW16I have inductive suppression or other circuit protection on them ?   Obviously "a relay failed to open as expected" suggests a stuck or welded contact in the relay.

How was the system "stop" eventually accomplished ?   Did it happen later than expected, or not until another mechanism like E-Stop was used ?

What version of the Redundancy firmware bundle does this system run ?   Have you verified that all the modules in the system are correctly loaded with firmware that is compatible with that bundle ?

[GuyMiller 1]

"RIO" - EtherNet/IP, through a 1756-EN2T module.

Yes, I think a broadcast storm occured causing a network issue on this network.

Yes, the disqualification and the failure to function as expected occur at the same time.

I don' believe the circuit have suppression or protection on them.

Disconnect was pulled to stop the device.