Ethernet message

[lccllc 0]

I am going to be starting a project with two L61, one will be the main plc the other will be a partner plc to watch the main plc. I will have to send status data from one to the other also i/o data from one to the other for redundant monitoring of the I/O. I think it is better to send one lump sum of data rather than sending milti messages with part of the data, to increase the speed of the communications. Am I correct or not, I would also like to know, if there is a chart or something that will explain the effect of sending large sums of data through a MSG instruction. I will be using Ethernet for the data MSG. I will also have four drive on the network, so there could be alot of traffic on the network.

[BobLfoot 302]

Is there a reason you're not using the SRM Module and the redundant features of Logix 5000? Hopefully my good friend Paul will see this post sometime and grace us with another of his awesome math equations for network usage and such. But until then yes a lump sum is better than individuals although I seem to recall that the more current versions of Logix {16/17} optimize comms somewhat anyway.

[Ken Roach 194]

It is generally much more efficient to send data in a larger block than to send multiple smaller blocks, and not just because of the relatively large overhead involved in TCP/IP transmission. The controller's Ethernet interface and backplane buffers handle single larger blocks more efficiently than multiple smaller blocks because the resources allocated to each transaction are the same. That being said, the larger question needs to be asked: Since you are using ControlLogix, why not use the ControlLogix Redundancy system ? All the data transactions and network connection backup mechanisms are already there in the Redundancy Modules and the Redundancy firmware.

[lccllc 0]

I guess I should not use the term redundant PLC's with this question, I should of said the partner PLC is being used as a watchdog to the Main PLC. The main PLC is going to be a 1756-L61S and the partner PLC will be a 1756-L61. This project needs to be very safety minded, and the partner plc will also be doing most of the faults and diagnostics on the system. Ether way the posts pretty much answer my question. I do hope Paul sees this post, thanks Bob and Ken for the posts.

[paulengr 44]

So you've got 3 PLC's! You really do have 3 of them. A 1756-LxxS PLC is actually 2 identical Lxx PLC's tied together with a comparator circuit that verifies that both are operating in lock step and producing the same identical result. If not, then it shuts everything down and sends the system into fail safe mode. This is what makes the PLC a safety PLC. The exact opposite intent is created with the redundancy modules. I'd forget about actual bits per second type limitations if I were you. In practice, the limitations seem to be first on connections, second on packets per second, and third on actual bits per second. It is of course always possible to find degenerate cases though where you could run out of packets per second or bits per second first. Ethernet (and internet protocol) have no inherent packet length limitations. It is purely an implementation detail. If you somehow manage to exceed the allowable length, then a packet can be fragmented into multiple packets and then reassembled. In terms of efficiency of course this is not a good thing. For all practical purposes, a packet length around 1500 bytes is usually what is considered "the sweet spot". In addition, there is some overhead involved in the synchronization and coordination activities that go on with a packet, and traffic scheduling (prioritization) and all other algorithms inside an Ethernet switch are done on a packet-by-packet basis, not in terms of bits. Switches are non-discriminatory...skinny or fat, packet size makes no difference at all except if it's a "giant" size packet that has to be fragmented. So taking these two into consideration...making a packet as big as possible is the best approach. Although you could definitely create a MSG command that exceeds the "sweet spot", the Logix processor automatically detects this and chops up the MSG command into multiple chunks if this happens. Doing this end-to-end is not so bad compared to causing a switch to fragment a packet. Now to some considerations... For a single MSG command to write some data, there are actually 3 packets that get created. Roughly this is what they are: "Hey, I have some data to send" "OK, send away" "The data is XXXX" For a read, there are 2 packets. It goes more like this: "Hey, can you send me XXX?" "OK, the data is XXXX". In addition, TCP (these are all TCP messages) requires an acknowledgement packet. If there is already an outgoing packet in the opposite direction, then the acknowledgement is piggy-backed onto another TCP packet. If not, then yet another packet must be sent. Thus in the above scenarios if you don't have any other traffic ongoing, add one additional packet of at least 20 bytes. Up to this point, these are all TCP messages. TCP headers are 20 bytes long or more. This does NOT include any CIP overhead which is another 40 or so bytes. Then you've got to add about a dozen bytes plus the number of characters in the tag for the non-data messages. So figure at least 60 bytes plus the length of the data (DINT=4 bytes, BOOL=4 bytes (except if embedded in a UDT, then it becomes 1 bit) if we're talking about just the data packet. If you get over 1500 bytes as I said then you start adding packets but this should give you a rough idea. For producer/consumer, things are much different. It goes something like this: "Can you send me updates on data XXXX?" "OK" "Update #1: XXXX" "Update #2: XXXX" "Update #3: XXXX" The first two messages are sent over TCP. The rest go out using UDP and a very abbreviated CIP header. The total overhead on these is roughly 20 bytes because all the setup is already done. The UDP packets are minimalist in nature because producer/consumer traffic is really just taking advantage of the same communication mechanism in Ethernet/IP that is intended for I/O, and I/O is designed to be low overhead. Let's set aside I/O for a minute because there are some additional optimizations that can be done with that. When you set up producer/consumer transfers between PLC's, the data must be in the controller memory space (not program memory spaces). If you set up individual data items such as Tag_A, Tag_B, Tag_C, then each one will have it's OWN update packet, creating dozens of small packets. This chews up memory (about 1200 bytes/item) and chews up bandwidth. If you create a UDT or an array, then the entire object (array or UDT) is treated as a single data item. This is MUCH more efficient. With I/O, there is an even easier and more efficient way of doing things. Set up the first PLC just like you normally would. On the second PLC, simply add in the SAME set of I/O devices. However, change the "comms format" to "Listen-Only". In this mode whenever the I/O device sends an updated packet, it will broadcast a multicast packet. The switch(es) will automatically clone the packet and BOTH PLC's will receive identical copies of the packet at the same time. This works even if the I/O card is local to the first PLC but has to be replicated or broadcasted out via a ControlLogix backplane such as if the second PLC is in an entirely different backplane. The difference between the regular input/output modes and "listen only" is that the "owner" of the I/O card is responsible for sending any outputs such as configuration data and output data on output cards. The PLC that owns the card MUST be online or the "listen-only" PLC's will not receive any data. With input cards as long as the configuration data is identical on all PLC's involved, you can have multiple owners. With output cards, since the source of the "output data" would be ambiguous, only one owner is allowed (all PLC's except one must be listen-only). If you use rack-optimized I/O configuration, then all of the rack-optimized I/O data will be automatically aggregated and sent out as a single packet. This option is only available for digital I/O cards. If you use "direct", then each card generates a separate packet. In terms of packet counts, figure on a maximum of about 8000 packets per second (leaving yourself room for overhead and RS-Logix communication) for all of your I/O activities. Each MSG is going to spawn 4 packets for a write, or 3 for a read, worst case. In the case of producer/consumer inter-PLC traffic, there is one packet per data item. In the case of I/O, you get one packet per direct card, and one packet per rack if you have any rack-optimized I/O. There are configuration messages and other traffic but these are part of the "20% overhead". Now, you also have to be acutely aware of the number of connections you are chewing up since in practice, connections trump actual raw packets per second in most cases. The processor itself can handle a maximum of 200 connections (as per Logix manuals). A 1756-ENBT can handle 64 TCP/IP connections (1 TCP/IP connection per device being communicated with), and 128 CIP connections. Double these limits for the 1756-EN2T. One point of confusion that I haven't satisfactorily cleared up for my own understanding is that it appears that by turning caching off on connections you can multiplex these connection buffers somewhat, and the manuals seem to refer to the ENBT/EN2T cards as "expanding" the connection limits of the processors. So I'm not entirely sure whether the 200 connection limit refers to the processor itself vis a vis the backplane, or refers to a hard limit on all data paths to/from the PLC. If you get anywhere near these limits, then I'd certainly be talking to my local AB rep and/or TechConnect person to find out the answer to this question. To date, I'm still very much in the dark as to what constitutes a connection (of the different types) how to count them. The more AB tries to explain it, the more questions I seem to have. Best I can say though is that each MSG instruction is going to be using at least 1 TCP/IP connection. It will not be using a connection if you use unconnected messaging, otherwise it uses 1 CIP connection. Each producer/consumer data object uses 1 CIP connection. Each rack with optimized I/O uses one CIP connection. And each direct connection I/O card uses one CIP connection. This is all pretty much covered in the AB ControlLogix manuals, especially the Ethernet/IP dedicated manual. Be aware that the fastest way to cause mass chaos in terms of Ethernet/IP is to use all dumb switches and not to have an IGMP querier. This turns all multicasted Ethernet/IP traffic into broadcast traffic. If you have any PowerFlex drives (with 20-COMM-E I/O cards), these can only handle something like 500-600 packets per second. The drives will be totally overloaded and your Ethernet cards will CRASH if you use dumb switches that are not IGMP-snooping capable as a MINIMUM. If you want to avoid this and you don't want to spend a lot of money, I highly recommend that you invest in either a Hirschmann RS20 series switch or a Cisco 3750 series switch, both of which have the capability of being IGMP queriers or IGMP backup queriers in the event that your main (usually router) IGMP querier becomes disconnected. In addition, I highly recommend that all your switches have IGMP snooping at a bare minimum. The Hirschmann SPIDER series and the low end N-Tron switches both have IGMP snooping capability and make excellent DIN rail mount switches for I/O. I have worked with these. I have not worked with them, but the switches from Moxa, Advantech, Sixnet I/O, and Garret also sound promising. I have a couple friends that swear by Phoenix Digital switches but I felt that they were overpriced (and so was Cisco). I do feel that at least Allen Bradley's dumb switches are fairly reasonably priced if you want to stick with the AB brand name. Be aware also (since oyu are considering safety aspects) that you need to be careful to limit the packet per second rate to each device, turn on broadcast storm protection, and I'd highly suggest either putting your I/O (and PLC) traffic on separate physical lans or else VLAN'ing them so that they are on separate logical LAN's. With both teh VLAN's and the individual ports, I'd also suggest considering setting up QoS (traffic priorities) to ensure that your critical data (I/O, inter-PLC communication) is protected and will also have bandwidth gaurantees. If you do this stuff, it will make Ethernet/IP safe and secure...in fact, you can often even make it safer than any other network especially if you buy ring switches to allow deterministic I/O failover in addition to deterministic I/O as long as you mind your packets per second and design accordingly.

[lccllc 0]

Paul that was a great answer, thanks for explaining it to me. I did not even think about setting the I/o of rack one for listen only. I am going to be using a phoenix contact managed switch, and all non local I/O will be on a devicenet network, including safety i/o. I have done this on another project and it seemed to work well by seperating the hmi, plc and drive trafic from the non local i/o traffic. Thank to all that have helped, this forum has some of the best people on it. Thanks again, LCCLLC