Losing comms on CLX Redundancy swap over

[jbord37 0]

Hi everyone, I have an L72 CLX Enhanced Redundancy (just upgraded to latest Red FW package 20.055 rev4 in all modules). It is a 4-slot system with L72, EN2T,CN2,and RM/B modules. This PLC has been on its own network of subnet 192.168.115.0 for a year and we have tied it into a newer side of the plant network which uses a 10.75.0.0 subnet and incorporates a vlan2 of subnet 192.168.115.0 (for communicating to the old side). The gateway to access new network has been set in all of the old subnet devices, and we can communicate (PLC to PLC MSG, OPC from Industrial Gateway Server) flawlessly until a redundancy change happens. I have checked the switch, and the arp table updates with the IP/MAC address swap over, but until i issue a "clear arp" command from the cisco IOS CLI, the communication between subnets cannot get through. This happens whether the vlan 2 gateway interface (192.168.115.1 255.255.255.0) resides on the core switch (Cisco 3750) or on the switch directly physically interfacing between the subnets (Cisco 3010- half of ports are vlan2, half are access ports). There are several VFDs on the 192.168.115.0 subnet that are all communicating with the processor in question, and they don't lose comms on a role swap, only devices in the 10.75.0.0 subnet. I have not been able to get a wireshark yet, which I'm in the process of doing but I will need some help from our IT people. Any help or experience with a similar issue would be greatly appreciated. Jason

[MrAutomation 20]

I would suggest asking this at http://serverfault.com/. This looks like it's a purely network related question, and the guys over there are most likely better suited to answer your problem. Also, it's not entirely clear what your network infrastructure is exactly. Are you using Spanning Tree? REP? What's the physical and logical topologies? What exactly do you mean by redundancy change? Where is the break? How is it introduced? If you ask your question at ServerFault and provide the extra details above, you'll probably have better luck. Hope it all works out for you!

[jbord37 0]

Thanks for the tip, I am not familiar with that site, do they have people knowledgeable about ICS? We have an IT professional group we have contracted to help us with this and unfortunately, due to their lack of automation experience, have not been able to help much. Your infrastructure question hits home as unfortunately the topology is a bit of a mess in some sense. What I have is a fiber ring set up using REP topology with four Cisco Stratix 8000 switches on the 192.168.115.0 subnet. On the 8000 which we've designated as "master" where the PLC resides, we have, plugged into one of the ports, a Cisco 3010 switch with a 192.168.115.0 subnet vlan interface serving as the gateway, and a default vlan 1 interface in the new 10.75.0.0 network. This 10.75.0.0 network is a separate plant wide ring using spanning tree with the links between the ports trunked to dot1q. link-type point-to-point. There is some fuzzy area here as even though spanning-tree is disabled according to the Stratix config, I'm not sure if the vlan 2 access ports on the Cisco 3010 should have spanning-tree like the rest of that ring. This is unfortunately the kind of situation you get into when one engineer specs stratix switches and they are set up using REP, and the engineer on the next phase of the project demands Cisco switches and also spanning-tree. The redundancy change can be introduced by removing the ethernet cable from the En2T module on the primary rack and forcing the secondary to take over. We have seen some other strange things lately though also cause issues accessing the PLC, even though nothing else in the 192.168.115.0 subnet has any issues with us continuing to ping it after we lose PLC comms. Maybe someone else has had this issue when integrating subnets, but due to the resources and pressure I have here, my next best option is to replace the 4-slot racks with 7 slot racks and put an EN2T module dedicated to the new subnet in an empty slot. I have a feeling this issue won't go away though if there is another flaw somewhere in the system. Thanks, Jason

[MrAutomation 20]

Fun. If I have a minute later I'll draw this all out on a sheet of paper and try to make sense of it. If i understand correctly,you have a redundancy enabled on the PLC cards, but one port is connected to the old 192.168 REP network, and another is connected to the new 10.17 spanning tree network? If that's the case, I can't imagine it really working, and on top of that, it's not necessary. Just have them both on, all the time. Different networks, VLANs and IPs. No need to have any fail over there. But I didn't have time to really read it, so I may have misunderstood. I'll try and revisit this later. Good luck!