Overloaded Ethernet IP network (?)

[RFurey 0]

I have a process that uses an Ethernet IP network for the control system. I'm having problems with it that appear to be associated with too much traffic, at least that is what tech connect says though they haven't been able to help too much. The system (Qty 1) CompactLogix PLC 1769-L35E (Qty 3) Ethernet switches (daisy-chained together) (Qty 3) Panelview Plus 600 (Qty 3) Ethernet Adapter for remote I/O, 1734-AENT Each adapter has two output module 1734-OW4, three Analog module 1734-IE2V, three TC module 1734-IT2I. (Qty 1) Desktop PC running an RSView32 project (200 tags or so) (Qty 1) Laptop running the same RSView32 project while I'm in the process of modifying it (300 tags or so). (I'll eventually move the modified project to the desktop and remove the laptop from the network.) I also usually have RSLogix running too. The problems A) The two RSView32 projects do not appear to be able to run at the same time without creating OPC errors. One type of error in the RSLinx DDE/OPC Communication Event Log says "CIP Messaging Error as a result of a CIP connection being dropped…" The second error type says "ControlLogix Optimized packet being reinitialized 13c50e8…" B) One of my Panelviews also has repeated communications problems (I neglected to note the exact message on the Diagnostic screen). If I turn it off/on the problem goes away for a while. Questions 1) Does this look like a particularly "busy" EthernetIP network? It's my first one that has more than a couple devices. I was under the impression that an EthernetIP network could handle many devices. 2) Should I be able to have two RSView32 projects running on the same network and accessing the same PLC at the same time? 3) Are there things I can do to optimize RSLinx or the PLC or RSView32 to make these communication problems go away? Any help or advice would be (very) welcome.

[newpageboba 6]

Just some quick advice. The Automation Fair just had some good discussions regarding this and Tech Support should be able to help. Check RA Knowledgebase for the following topics: Ether/IP and also Logix connections. The thing to check is Processor connections and Multicast traffic. Other members of this forum will also have some good advice Good luck

[Eddie Willers 2]

The limitation in an EtherNet/IP network is almost always the port on the controller. The CompactLogix is meant to be a small machine controller, so it doesn't have a massive communication coprocessor. In your excellent description, you did good job of describing about "how much" data you're requesting from the controller. The second half of the equation is "how often ?" Use the diagnostic webpage of the CompactLogix to determine how many Class 1 packets per second your I/O connections are taking up. You can also figure that out by adding up the connections in the I/O configuration in RSLogix 5000 according to the equations in the EtherNet/IP Performance and Application Guide. Investigate the update rate of your PanelView Plus displays. Do the same with your RSView32 scan classes. communications timeslice value. Default is 20%. Five simultaneous HMI sessions qualifies as a "high bandwidth" requirement in my book, so I'd increase the timeslice to 30 or 40%. Put in GSV instructions to monitor the Module connections of the POINT adapters to make sure the CompactLogix isn't dropping those I/O connections (it shouldn't be, but it's possible) when the RSView sessions are hammering on the port.

[Ross 0]

I'm not an expert in Ethernet IP but have had to deal with similar communications issues in controllogix before. Here are some things I've done that have helped. 1) If you are running a continuous task set the system overhead timeslice to 50%. Be careful your scan time will go up and vary more. Alternately you can move everyting to a periodic task set to something like 50 mSec if your process can tolerate it. 2) Make sure your Panelview and RSView tag update rates are 1 second or greater. The less often they ask for data the less traffic on the network. 3) Put all your PLC data in continuous blocks of data AKA arrays. RSLinx optimizes data based upon where it is in memory and if your reading data in the same general area it will read the whole block once and parse out what it needs. 4) Buy RSLinx gateway for the HMI PC (sorry) and set one of the RSView projects (Laptop) to use the remote OPC server. That way only one of the RSview projects is pounding on the controllogix for data. 5) Get ethernet switches that are managed and set them up so that they will only communicate to the ports they need to. Make sure the Switches are 10/100 so that the connection between switches is 100 MB. Here's where IT can help out.

[Contr_Conn 4]

You already got good starting hints from posts above, I just want to add few things: It does not matter how many devices you have, what matters - how much data and how fast you trying to get. I can easily overload PLC just by using few I/O modules out of single 1734AENT. Ethernet/IP hands-on lab at the Automation Fair gave example of the largest single Ethernet/IP installation: 1600 nodes. With single compactlogix controller you probably will not get fast updates in any case. You need very carefully analyze all data sizes and transfer rates. How to do it? - see above. In reality this should be done at the design stage, it is little too late now. Collect all data and post it here, and you should probably call techsupport again with these numbers as well - they should be able to analyze it and provide solution.

[Gerry 12]

Why do you have ??? Surely one would be sufficient? I think it must add delays having all traffic negotiating its way through 3 switches. Check the RPI settings for your I/O. They default to low values 2 or 5 msec. Generally, 20 msec or higher is sufficient.

[ddevins 0]

All the above help is good. I recently attended an ethernet seminar at local AB distributor. It is amazing what a difference managed enet switches can do to solve traffic problems. Easy to config. and reduce unneeded traffic. Hope this helps.

[BobLfoot 138]

Not nearly that large but did do a ControlLogix L61 application with six 1756-ENBT modules and 136 XYCOM HMI Panels on ethernet all accessing the L61.

[RFurey 0]

Thanks for all the info. It seems like I was sold on EthernetIP without fully considering its downside. I was under the impression when I started that it was sort of a plug and play network that was simpler to use than Devicenet. I also thought that I had lots of room for expansion with the default settings before I hit the wall, so to speak. I guess I have some catching up to do. I've increased the timeslice to 50% and I've increased the Panelview update rate to 1 sec. Currently the frequency of CIP errors have declined but not gone away entirely. Since it appears that the one hardware improvement I can make is to replace the switches, I'm looking for a greater than 9 port managed switch to replace my three unmanaged switches. Any recommendations? Is there anywhere I can find a step by step methodology to map out an EthernetIP network to prevent this sort of thing from happening again?

[TimWilborne 103]

On thing that gives Ethernet/IP a bad rap is a lot of people think just because it has an Ethernet Port and is made for a PLC it is Ethernet/IP. This simply is not true. The culprits in your case are the Panelview Plus and PC running RsView. These are not Ethernet/IP devices. I know everyone wants the latest and greatest but in contrast the Standard Panelview is Ethernet/IP compliant. Then you go to the hardware you are connecting all of this with. There is a big difference between a $20 8 port switch you buy at Staples and a $200 8 port industrial switch. But from what I am reading you are not having any trouble with the communications between your Compactlogix and FlexIO so we will hold off on that. Can you post a program and give us a bigger picture of what all is going on in the Compactlogix and your network. Also some screen captures of your task monitor of the Compactlogix may help find exactly what the bottleneck is

[Ken Roach 170]

I always hand out at least three documents when I do an EtherNet/IP seminar: EtherNet/IP Media Planning and Installation Guide, publication # ENET-IN001 EtherNet/IP Performance and Application Guide, publication # ENET-AP001 EtherNet/IP Modules in Logix 5000 Control Systems, publication # ENET-UM001. I also try to include a brochure from Hirschmann or NTron for industrial switches. I have a strong dislike for salesmen and even competitors who pitch Ethernet for industrial control networking as miraculously cheap and easy. Ethernet's speed may mask a multitude of design and installation mistakes, but it's still a network and can still give you grief you can't see without tools. Edited 30 Oct 2006 by Ken Roach

[BobLfoot 138]

Great References Ken - Sounds like you will be authoring the "AB Ethernet IP Bible Powerpoint" soon. You know like that DeviceNet Bible Powerpoint and CD that came out a while back. Think about it we'd all benefit from you putting that together.

[TimWilborne 103]

I like how smoothly you added that to Ken's list of things to do

[Peter Nachtwey 13]

I don't think the net is overloaded. I think the there just isn't enough processing power. I have seen control logix networks with many of our controllers on one network and the data demands were must have been much higher than in the OP systems. Just because the Ethernet is 10/100 doesn't mean that it can all be processed at that rate. I don't know anything about the Compact Logix Ethernet but I bet it isn't using the same processor as the ENBT. In addition the Compact Logix Ethernet must aribtrate for the PLC bus access and that will slow things down.

[rammin48 1]

I have implement several large Ethernet I/p projects and had some issues as well. You need to use managed switches, also look at who is sending and requesting what data. I have found that not only on AB but on other PLC's you have to be aware of data amounts and how often. Look for the bottle neck. Try not to send and recieve from one device. Spread the comms out. I have had to move the RSView stations onto a secondary network or utilize a gateway to reduce traffic. Making minor adjustments in packet size and comms instructions can have huge results as AB has a tendency for collisions. Also various mixes of platforms can impact updates as well. The managed switches can have a great impact with the lease amount of programming changes. Watch the number of MSG instructions executed at a time. Too many of them will slow down communications at a single point. Again a managed switch can help you find the trouble makers. This is not an Ethernet specific issue. I had to do this with DH+ before. Good luck. Keep fightin!

[BobLfoot 138]

Excellent Point - I recall when we spent weeks trying to streamline code and use smaller memory chips to save weight and money. Now we throw away tons of memory using full bytes for a single bit tags. The elegance of design is lost on many of the "point and click programming school".

[RFurey 0]

As I've move from simple PLC applications consisting of a single PLC (Micrologix or SLC500 with maybe one Panelview or RSView32) to larger systems consisting of a Control/Compact/Flex-Logix controller, multiple remote IO, EthernetIP, more than one Panelview Plus and/or RSView32, I find myself chasing more and more "ghosts" - ghosts being problems that occur for no apparent reason and are exorcized by resetting or reloading or rebooting or re-something. Example: a) Today I had a Panelview Plus drop out of an Ethernet network (I couldn't even ping it) after a power down/up. The two other Panelviews in the network came up fine. I finally got it running again after following some steps from Tech Connect which consisted of resetting the default communications, resetting the Panelview and then re-transfering the program. Why the Panelview crapped out in the first place remains a mystery. b) A short time later a different system (Flexlogix, RSview32) died when the Ethernet IP daughter card in the PLC experienced a major fault. Unfortunately this occurred while running an (expensive) process. Unable to power down, I elected to pull and reinsert the card, hoping it would reset communication. It did but the processor then acted "oddly". I hooked up my laptop and found that the program in the PLC was gone! I re-downloaded the program and it is again working ok (for the time being). Is this sort of routine normal for the more complex control system? If so, is it Allen-Bradley-centric, or do you guys deal with these issues all the time? There was a time when I would put together a PLC system and forget all about it. I rarely (if ever) had a problem with them. These days I can't go more than a couple months without some sort of hiccup that can not be remedied at the tech/electrician level, costs us money and defies explanation (Tech Connect - "Hmmm, that's weird…"). Am I in the same boat as the general controls community or am I an outlier who may be doing something wrong?

[TimWilborne 103]

RFurey, I merged the two topics and deleted the cross talk between the two topics. Let's see where it goes. If you and others feel they are unrelated issues then we can split them back up. Can you give us some details of your network including hardware used? A picture would be nice but descriptions of how this get from point A to B will be a start. Also might want to check all of your grounding

[RFurey 0]

[TimWilborne 103]

It could, give use some part numbers and basic layout Also your screen capture is of the task properties. You need to look at the Task Monitor. It came on your CD and I think it gave you the option of installing it. If you don't have it download it from the link below. http://www.software.rockwell.com/download/...tor_v2_0_05.zip Also you might download the Ethereal Network Analizer. It gives you every spec of information about traffice going across. Even though I think it is in the physical media or grounding it may also give some clues. The only problem is we will need a network guru like Ken Roach or Contr_Conn to interpret the data. All I can tell you is if it is good or bad http://www.ethereal.com/

[Ken Roach 170]

Three 1794-AENT adapters each with two modules that are part of the 20 ms Rack Optimized connection, and six modules that have direct connections (analog and thermocouple) at 50 milliseconds. The EtherNet/IP Performance application guide tells us that the number of packets per second handled by the CompactLogix will be (2 x Connections) / RPI. (2 x 3) / 0.020 = 300 packets/sec (2 x 18) / 0.050 = 720 packets/sec This total of 1020 packets per second is well under the maximum of 4000 packet/sec that the 1769-L32E/L35E port can handle. The RSLogix 5000 Task Monitor can help a little with figuring out how hard the Ethernet port is actually working (CPU-wise) to tell us whether to focus on the RSView / PV+ performance or on the switches and cabling. It would be interesting to run an analyzer at the computer side to see how much UDP Multicast traffic the PC's network card is having to discard to get it's ordinary TCP/IP data through the port.

[TimWilborne 103]

Are the switches daisy chained so the switches are closer to devices or because there was not enough ports on one switch? A managed switch with port mirroring would be best but is not an absolute necessity I wouldn't be too quick to replace those B&B Switches. I have never used them but they look like a good product. Not managed but...let me look a little more into the daisy chaining. I have done it but never where I would have to transmit I/O messages across the switches

[TimWilborne 103]

Also what type of cabling are you using? Standard Ethernet cable or a Shielded Ethernet cable such as Beldin 7919A If you are using Shielded cable how are you terminating the shield? What distances are you cables going?

[RFurey 0]

Standard CAT5E cable. Max cable length is less than 50 feet for each cable. The cables are either run in conduit with 24 VDC or the cables are run alone. In each of the Panelviews the Ethernet cable shares 10 feet of swing arm tube with a 120VAC E-stop circuit. In the Compactlogix case, the three switches are in the same enclosure and were added as the system grew. (For simplicities sake, I'd like to concentrate on the Compactlogix system unless it becomes readily apparent that the Flexlogix system is suffering from the same problem.) Task monitor data TASK_MONITOR.doc. I really appreciate the help.

[RFurey 0]

I switched the three Ethernet switches to one managed switch this morning (N-Tron 516TXA). According to the N-Tron tech, the switch was supose to improve communications without any configuration. No change. The same problems still persist. On to Plan B, once I figure out what Plan B is......