Sign in to follow this  
Followers 0
zenerdiode

Issue with Allen Bradley PLC Modbus TCP IP

8 posts in this topic

Dear Forum,

I am working at a Gas Processing Plant which is running Emerson Delta Distributed Control System (DCS). The Control System is communicating with over 24 3rd party Packages in the field over Modbus TCP/IP using its VIM Module. We have implemented redundant communication with most of the packages so we have a primary switch and a secondary switch. We are facing a communication issue regarding an Over Head Compressor (using Rockwell Allen Bradley PLC) and Instrument Air Compressor by CAPS Australia running XE Controller (Dedicated Controller). The issue is that when the Over Head Compressor is plugged into the VIM Network (Network in which all 24 3rd Party Packages are plugged), the Instrument Air Compressor **stops communicating**. We can't even ping the compressor. At times the Instrument Air Compressor does reply but with TTL of over 30ms. When OVHD Compressor is plugged out, Instrument Air Compressor resumes communication.

I have attached wire shark capture of the network with and without over head compressor. When the OVHD Compressor is plugged in, the capture shows heavy traffic being generated by its PLC (IP address 192.168.1.60). Another physical observation is that before the OVHD Compressor is plugged in the VIM Network, the NTRON Switch (Ethernet Switch) in OVHD Compressor panel shows static lights, unlike lights of a normal healthy network. When the OVHD Compressor is plugged into the VIM Network, all the lights of switches in VIM Network which were blinking previously also become static. Do the static lights mean that the network is being choked due to heavy traffic?

Below are links for the Capture, IP Scheme and Architecture of Over Head Compressor.

https://www.dropbox.com/sh/ar7ss8zzoo96g37/AAADguRpUDtdOnmN3xQX-Wlpa?dl=0

Please also note that OVHD Compressor is using following IPs

192.168.1.59  PLC (Internal PLC IP)

192.168.1.60  PLC (Internal PLC IP)

192.168.1.61  HMI (Internal PLC IP)

192.168.1.62  RedLion device for DCS Communication

I would really appreciate if anyone would help me trouble shoot this issue.

 

  

Share this post


Link to post
Share on other sites

Break the situation down...your Modbus TCP/IP over Ethernet network is functioning, until you plug in the Overhead Compressor.  Is it possible the overhead compressor is bombarding the Modbus network with too much traffic?  Is it possible there is a duplicate IP address?

Which Rockwell/A-B CPU is being used in the Overhead Compressor?  A-B PLCs are known for significant broadband traffic in standard configuration.

 

Share this post


Link to post
Share on other sites

@kaiser_will Please if you can, go through the dropbox link, I have uploaded the architecture drawing of Over Head Compressor which contains complete information of the PLC and Cards used. And also the IP scheme of the complete network. We have already verified, there is not duplication of IPs in the network. We are also following up with the Vendor and have raised exactly the same concern. Maybe they would review their configurations in PLC for modbus. Analysing the Wire Shark capture, it clearly shows that the AB PLC is actually flooding the network with its packets.

Share this post


Link to post
Share on other sites

It appears the Over Head Compressor, with its A-B PLC, is the culprit for shutting down shortly after it is connected to the controls LAN.  The Ethernet switch that this machine is plugged into, it is a managed or unmanaged switch?  Ideally one wants a managed switch that offers IGMP Snooping; enabling IGMP snooping allows multicast consumers to register for multicast traffic, preventing multicast traffic from reaching nodes that are not meant to receive the traffic.

Have you gone into the PLC processor to retrieve its connection history - this history is the table of events that can guide you to the root-cause?  Which model of A-B CPU are you trying to integrate?  Are you using Produced/Consumed tags in your A-B application?  Do you have fast I/O update blocks in the A-B application?

Refer to page 29.  A-B pushes their Cisco-designed Stratix switches (expensive, but worth it); Hirschmann are excellent and suggested by A-B, also.

http://literature.rockwellautomation.com/idc/groups/literature/documents/rm/enet-rm002_-en-p.pdf

Do research on "logix ethernet network traffic", or similar.  Ethernet is an open network protocol; Ethernet/IP, using Ethernet as the backbone, is not deterministic.  If the network is bogged down, CPUs can go offline (as in your case).  By anticipating the Logix CPU's behavior, one can adapt it to the network to limit network traffic and keep everyone happy.

Share this post


Link to post
Share on other sites

@kaiser_will Thanks alot for your help. We disabled "Supervisor Mode" in the Ethernet Card settings and now the behaviour of the system is normal, the lights of the switches are blinking and the Instrument Air Compressor which was not pinging before, is pinging and giving data on Modbus TCP/IP. 

Edited by zenerdiode

Share this post


Link to post
Share on other sites

I'm glad to hear you got it sorted out !

I used to do this sort of analysis and troubleshooting for a living, so I'll mention that there are definitely professionals who can solve this sort of problem quickly and definitively.   We just aren't cheap to hire.

The layout and the Wireshark traces are conclusive;   this was a Device-Level Ring (DLR) configuration issue.

Rockwell's two-port Ethernet modules, including the 1756-EN2TR that you are using, have a special feature that they can run on their dual ports called "Device-Level Ring".   It's a very fast-healing ring protocol developed by Rockwell and Cisco.   Many other vendors have similar protocols, many of which are proprietary or semi-proprietary and are all improvements on the old-fashioned Spanning Tree Protocol.    DLR allows Rockwell devices and similarly equipped switches to form a ring topology that can heal a broken cable by re-routing data, and do it fast enough to not affect high-performance protocols like CIP Safety and EtherNet/IP.

Your layout shows two 1756-EN2TR modules in the ControlLogix chassis, each connected together in a daisy-chain (not a ring, as far as I can see).   But the other devices are not DLR-capable.

N-Tron makes great switches, but they have their own ring protocol called N-Ring.   They do not support DLR.

The "Enable Supervisor Mode" selection checkbox for the 1756-EN2TR makes them send out DLR management packets, two or three per millisecond.   Since the N-Tron switches don't know how to interpret these, they send them out to all ports.   You were very likely creating a broadcast storm, which is why the port traffic indicator LEDs on all of the switches were lighting up so fast they appeared to be on steadily.   This may have been aggravated by the redundancy system you use, however that's set up.

You're going to want to review the switch architecture and test to be sure it works the way you expect.

 

 

 

Edited by Ken Roach

Share this post


Link to post
Share on other sites

For a little light reading, you might enjoy the EtherNet/IP Embedded Switch Technology booklet:

http://literature.rockwellautomation.com/idc/groups/literature/documents/ap/enet-ap005_-en-p.pdf

In your case, the "Enable Ring Supervisor" had to be done intentionally while online with the controller;  it's not the default setting.   So this is a case of somebody enabling something that they didn't understand, and it having a negative effect on the overall system.

Boy howdy, have I done that.   Nothing like watching a whole paper machine begin coasting to a stop when you hit ENTER.

1 person likes this

Share this post


Link to post
Share on other sites

We are having similar issues at a facility, and would you believe it is also a piece of vendor equipment on 192.168.1.60!

Ken, I work in Oil and Gas, nothing like seeing 2-4000psi product suddenly flare off to atmosphere when you hit ENTER....

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0