Sign in to follow this  
Followers 0
paulengr

Strange Ethernet/IP "blips"

7 posts in this topic

It's happened twice now. I've been running Ethernet/IP networks for a few years now and most recently started using them for I/O. We have a 1756-L61 with 5 remote I/O's. Two are remote racks. The other three are 22-COMM-E's (Powerflex 40's). At the end points, we have N-Tron "dumb" switches. These are connected to a Cisco 3570 switch. There are also two Wonderware servers connected via several more switches talking to the PLC through the same 3570. Well, we did. I substituted a Sixnet switch for the N-Tron at the PLC end of things simply because it's the first managed switch I could lay my hands on so that I could monitor the ports more closely at the PLC end of things. However, twice now in the past 2 or 3 weeks, all communications has ceased for about 10 seconds, and then reconnected as if nothing ever happened. The remote racks and drives are all programmed to fail safe when this happens (shut down). The trending data on the servers indicates that there was never a loss of communication between the servers and the PLC. All the port monitoring data says that all ports were OK with no loss of communication. Nothing in the logs indicates power failure (which if nothing else would be loss of communication). It's as if at the CIP level of things, the links were all severed and then recovered for no apparent reason. The really strange part is the server communication. I would narrow down the failure cause to either the Cisco switch, the N-Tron (now Sixnet) switch, the ENBT card, or the cabling between all of these if we lost communication with the PLC altogether. But during these events, I'm getting data updates with tags that are changing values at the servers which suggests that all the above hardware is indeed communicating. It's as if all producer/consumer traffic suddenly imploded in a puff of logic and then got regenerated. Has anyone ever seen anything like this with Ethernet/IP? I'm waiting for my license for RS-Logix Pro to get here because if nothing else, I'd really, really like to load up RS-Networx and poke around and see if there's any obvious configuration issues that were missed.

Share this post


Link to post
Share on other sites
All ethernet switches and hubs are not alike. I have learned, the hard way, that you should test them off line. N-tron is a solid product and reliable and my experience with sixnet from the past was not a good experience so I won't comment as I have not used them for years. The best way to trouble shoot is using a managed switch. Take a diagram of your network and look for bottleneck points. All devices should be configured for a fixed transaction speed and not auto-negotiate. The fixed speed should be the slowest device on the network. Place the managed switch such that you can monitor most of the transactions and look for the device that is generating the most requests. The switch is one place I would look first. Then have you performed any software upgrades? Look at the update rates on the devices. See if something has changed. I redownloaded the program to a L61 and lost the RPI settings so check those to make sure they haven't been changed or lost. This is general testing I have used. Some may or may not apply and without a hardware layout and configuration settings it is dificult to point you in a specific direction. There are some monitoring software you can get online that looks at packets and transactions. Some have costs. Good luck as AB ethernet is not easy to troubleshoot and may take some time and patience.

Share this post


Link to post
Share on other sites
I had a problem where when data got to a certain traffice level, the cisco switch would shut the port down momentarily. This was caused by Voice over IP settings on the Cisco switch. Their is nothing wrong physically with the network, just the amount of traffic. Did you have your Cisco firmware upgraded?

Share this post


Link to post
Share on other sites
I have no control over the Cisco garbage. The port shutdown garbage is the reason that Cisco cannot be used in an industrial controls network as far as I'm concerned, no matter how much AB tries to be their marketing arm. My next move is to bypass it. That being said, as soon as I turned on IGMP querying (guess what IT wasn't doing?), the packet rates dropped significantly. Apparently even the Cisco stuff was configured with IGMP snooping, just not generating queries. I have drives with 22-COMM-E boards which only have packet limits of about 500-600 packets/second. The rest is ENBT's which have packet throughputs up to about 5500 packets/second, and they're reporting loads averaging around 1100 packets/second. I'm somewhat concerned about the total bandwidth limits in the Cisco switch as they have everything VLAN'd properly as far as packet isolation goes but not bandwidth control/isolation. That requires a whole other conversation.

Share this post


Link to post
Share on other sites
I agree about the Sixnet device, but the above... NONOONONONONONONONONONONO. Unless you are one of the 0.015% of super-networking guru's AND have a solid reason, NEVER change a device from AutoNegotiate. That is a recipe for problems. For one thing, you have to guarantee that AN will be turned off forever on each end of the wire connection between devices. Switch dies? Someone replaces it? Leaves it default? Broken, damaged connection. Dropping in an unmanaged device where you can't set AN? Broken connection. The bad part, is having an AN mismatch may not show up for anywhere from minutes to months, but it will. Turning off AN will usually result (no matter how carefully you try to configure the endpoint devices) in the wire-mode being set forever to half duplex, instead of full. For modern devices (less then about 7 years old) turning off AN can also screw up MDX Autoswitching. 10 Years ago or so, when FastEthernet was first becoming popular, AN devices had some issues. That has changed.

Share this post


Link to post
Share on other sites
If you are getting an accumulation of physical errors like noise, or getting auto-negotiate errors, you're going to see that in the embedded diagnostics of the 1756-ENBT and the 22-COMM-E, so check there. Use a web browser to examine the status of the 1756-ENBT's I/O connections. If they all have the same uptime, that tells us something, because they all failed and re-started at about the same time. The web browser will also show you the CPU utilization of the 1756-ENBT. Because EtherNet/IP is a stack-based protocol, it is possible to just overload the module with traffic and have it drop I/O connections. That's one of the reasons I use ControlNet for I/O. It's possible for the Atmel backplane chip issue from a few years ago to behave this way, breaking one or more I/O connections at the backplane level and not faulting the -ENBT or the controller. Double-check for Atmel backplane bridge chips with -64 at the end of them and use the RA Knowledgebase document to check for the remanufacturing indicators. If you can mirror the port that the 1756-ENBT is using and capture the failure as it happens, you'll see whether the 1756-ENBT just stopped transmitting, or if it stopped receiving from the field devices, or if something else happened that caused traffic to cease to that port. I keep a 1756-L1 handy just to use the serial port to stop my Wireshark or TShark captures via ASCII and SerialKeys (there's a how-to in the Download section).

Share this post


Link to post
Share on other sites
At this point I was looking for ideas before I went crazy with Wireshark and waited again for weeks before it happens. That is of course the next best thing. Hadn't thought of doing this before and I left the L61 serial port open (don't really need it except for troubleshooting), so this might be a good time to try using it if I get a third outage.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0