Sign in to follow this  
Followers 0
sparkotronic

Ethernet Problems

24 posts in this topic

Hi all, yet again I'm looking for some AB help! In my workplace all our PLC's sit on an ethernet network. On the network there is 10 PLC-5's, 2 Control logix and 14 PC's running RSview32. We have a problem where one of our PLC-5 PLC's intermittently dropping off the network. We are a bit lost when it comes to networking and are trying to get a rockwell engineer on site to help us. I was just wondering if there is any software on the market that I could use to look at our network and monitor traffic, connections ect.? Does anyone use anything like this? Thanks in advance!

Share this post


Link to post
Share on other sites
Wireshark. It's free and it works

Share this post


Link to post
Share on other sites
Wireshark is often my first tool when doing Ethernet or TCP/IP troubleshooting. You can learn a lot from the Wireshark site itself, and there's a great user site called LoveMyTool.com (yes, I'm serious, that's the name). The best industrial protocol analyzer on the market is Frontline Test Equipment's NetDecoder. It's not for casual users; RA field support engineers often have access to it (if they do a lot of networking analysis). It's the tool I reach for when I actually suspect a protocol issue instead of a media/connectivity issue.

Share this post


Link to post
Share on other sites
A very good tool I can vouch for "WIRESHARK" has already been mentioned so I won't be-labor that point. Just don't forget that most of the newer PLC5's also have a web page interface which provides a wealth of diagnostic information just by entering the IP address of the PLC into your favorite browser.

Share this post


Link to post
Share on other sites
Thanks for the recommendations. I'm a total goose when it comes to networks/ethernet. I've had a wee look and play in my home network and it kind of baffles me? I think it might be beyond my meagre talents! I've also tried total network manager, this looks like it can tell me if PLC's are dropping off, but I don't know if the info you get is as detail. I'll have a bash with both tomorrow and see what I can find.

Share this post


Link to post
Share on other sites
You didn't specify, but if you are using RSLinx Classic (not Enterprise), and using OPC, RSLinx is probably your issue. I have fought with this for the last 4 years with no resolution. RSLinx just drops out and then will recover a minute or two later. Event log shows optimization and other comm message errors none of which point to a solution. Rockwell support just stopped answering the cases after a while. I'm dumping RSLinx for Kepware shortly. This might force me to replace very old PLC5's as the older firmware doesn't play nicely over ETH/IP and are not really happy using OPC, but since Rockwell can't fix it or even tell me why its happenning, its time to move on to something else.

Share this post


Link to post
Share on other sites
Hi all, I've had a bash with wireshark and had a look at the web based interfaces. What does it all mean....my head hurts and Rockwell won't send me an engineer. Seriously though, I haven't a clue where to start and I'm getting heaps of data, but it means nothing to me! I installed a piece of software that pings devices on our network and after a dropout, this shows no error. Could I be experiencing and actual PLC issue? Does PLC5 had a diagnostic page with a buffer(like Siemens S7). The problem seems to disappear just as quickly as it appears, which makes it a bit tricky to find, Thanks again for the help so far, I just need a torch now!

Share this post


Link to post
Share on other sites
Some screen shots of your Wireshark output would be helpful to this thread so we all could see and help....Or even your wireshark log files uploaded to this thread zipped up.

Share this post


Link to post
Share on other sites
PLC5 does have a diag page if its enabled, just browse to the IP. Are all the RSView nodes standalone? No centralized I/O data server? Could be connection overload.

Share this post


Link to post
Share on other sites
I've installed wireshark and started it. Do I need to do anything other than start a live capture? If an error condition exsists, will it "red flag" it? Once I have caught a drop out, I'll post the results here.

Share this post


Link to post
Share on other sites
Wireshark is great for analyzing data from one particular computer to a particular device. The problem you may have with Wireshark is that it mostly captures packets from the computer it's installed on to the device you are interested in (and you can use filters to weed through the junk). If you are trying to watch for a PLC that just "drops off the network", all Wireshark is going to show you is that the device is no longer sending packets out. If it really is dropping off the network, and depending how you have RSView set up (client/server or just clients), then you'll just see a break in communications with that IP address. What you might also have to keep in mind is the amount connections you are using. Each PC uses at least one connection to a PLC, and if you are using RSLinx, it can use many more. Depending on the firmware revision and network interface on your PLC, they support a limited number of connections. The best AB network cards right now support (I believe) 64 connections max. Your PLC-5 may be hitting your max connection limit or have a backlog for data. Try setting Wireshark to Promiscuous mode (Capture>Interfaces>Options>Capture Packets in Promiscuous mode) so that you read all the packets that pass by your network card. Set the Filter to ip.src == 192.168.1.XXX (or whatever your PLC IP address is), and watch the data go by. If you have intelligent network switches between your computer and the PLC though you may not capture all the data, the only way around it is to put an un-managed hub going from the PLC, then the PC that you are using Wireshark on needs to be plugged into that hub. If RSLinx is your culprit, then make sure you set up all the logging in RSLinx so that you can see what it's doing with the PLC when it drops out. You can also try to move the PLC to a different location (obviously put the right program on it) and see if the problem follows the PLC, if it does then you may have a hardware issue in the PLC itself.

Share this post


Link to post
Share on other sites
To add to what Ron said, I use a Phoenix 8 port switch that supports "port mirroring". This is really the BEST feature of any managed switch. If your PLC is connected to port 3, with a switch that can port mirror, you can mirror port 3 to the same port your PC is connected to that has Wireshark running. Very handy to have a switch that can do that.

Share this post


Link to post
Share on other sites
Hi Thanks for the info, The PLC in question goes into a managed switch, I'll have a look and see if it has port mirroring. I've heard linx mentioned a few times, this wouldn't affect plc to plc coms would it? I set up a recycling timer to pulse a bit every second and I'm now monitoring this bit on another plc so that if it drops out, it can raise an alarm. If it was a hardware issue, would the plc show any error states? Is there any subroutines I can put in to the logic to monitor for hardware errors? Thanks again for your help.

Share this post


Link to post
Share on other sites
Just a wee update. I finally got to see it drop off today. RSlinx had 2 errors 01E00204 and 0000005. When the PLC wasn't on linx I could acess the ethernet module (1785-ENET) via internet explorer. The LED's on the module points to it not sending any packets, but it has a network present. Does anyone have any idea about these error codes. Mr Bradley wants me to take out some sort of support package to acess their forums :(

Share this post


Link to post
Share on other sites
The 0x0204 error code is always an ordinary Timeout. Since you still have a Link indicator and you can get a HTTP response from the 1785-ENET, this sounds like an overload or malfunction of the PLC-5 controller or the 1785-ENET. The 1785-ENET original model supported only 10 Mb/s half-duplex Ethernet, and used an AUI port that allowed different co-ax, fiber, or twisted pair Ethernet connections. The modern 1785-ENET Series B has a built-in 100BaseTX port. The older modules could be upgraded to include Rockwell's modern "EtherNet/IP" protocol for communication with ControlLogix, and the Series B modules have that support out of the box. Determine and post the vintage and Firmware revision of your 1785-ENET and the PLC-5 controller to which it is attached. These modules had a limited capacity to reject traffic that wasn't intended for them, and in the days of hub-based Ethernet networks this sometimes caused the module to fault during a "network storm" condition. Because you aren't getting a fault (just a timeout), I don't think that's what is happening. How do you recover from this "dropping off" condition; do you need to cycle power, or does communication just resume ? The PLC-5 program should have a Status file you can define for Channel 2A, to look for errors from the controller's point of view. If you can intercept the traffic to and from the controller using an Ethernet tap or a mirrored switch port, a capture of a few megabytes of traffic using Wireshark could be examined by Forum members to see if anything interesting stands out.

Share this post


Link to post
Share on other sites
Hi The processor and e/net module are as follows: Processor: PLC-5/40 Series/revision: E/E.2 Ethernet Module: 1785-ENET Firmware Identification: 1785enet 2.50 01-Nov-02 We use Weidmueller IE-SW24-M Managed switches, version V5.1.8 We set them up as10Mbps Full-Duplex to our PLC 5 controllers and Auto Negotiate to other switches and PC's The E/net module is sitting on channel 3A, when I look at the Status bits (S2), I don't see any mention of Channel 3A The strange thing is that when it starts to happen, it does it at roughly the same time for 3-4 days the the problem disappears. I've added a couple of capture files, but they were taken when the system was ok, I do have some capture files the were caught under fault conditions, but the files are to big to post. I'll get myself into position tomorrow to try and capture some bite sized nuggets while or if it is happening Thanks again! capture5.zip

Share this post


Link to post
Share on other sites
The first thing I would investigate is the duplex setting for the PLC-5 interface. The 1785-ENET Series A and B sidecars have an AUI port to which you connect a transceiver (twisted pair, coaxial, or fiber). As far as I know that's always a half-duplex interface and the switch absolutely has to be set for 10 Mb/s half-duplex. I think it was the 1785-ENET Series C hardware that finally added a modern 10/100 Auto-Negotiating 100BaseTX port. A full duplex switch port versus a half-duplex transceiver will always generate large numbers of errors. To get statistics that are usable inside the controller program, you have to define a Diagnostic File, which is an Integer data table file that is dedicated to a specific communications channel. You specify this in the Channel Configuration for Channel 3A (right there at the top) and view the results in the Channel Status window. The only thing that jumped out at me from the Wireshark capture is that there are quite a few "TCP ACKed Lost Segment" errors originating from IP address 172.20.4.127 (a Dell computer). These are essentially analysis errors in which Wireshark sees the computer sending an ACK packet to a packet that Wireshark never saw. This can be due to capture issues (I don't know which port you mirrored) or routing issues and doesn't necessarily indicate a media or capacity problem. What is the IP address of the PLC-5 controller that is "dropping off the network" ?

Share this post


Link to post
Share on other sites
Sorry Ken, I should have includded that! 172.20.4.4 is the PLC that is dropping off 172.20.4.127 in the PC that is running Wireshark. I set the port mirror to mirror the port that the PLC is connected to with the port that my PC is connected with. I had read about the half/full duplex, I checked all our PLC5's and they are all set at full duplex, so I left it as that, I'll look into that today. If these are set wrong, would it stop the processor from transmitting? When the PLC drops off of Linx, I can still acess the Ethernet module via internet explorer. Cheers

Share this post


Link to post
Share on other sites
Be sure that you're looking at the properties for Channel 3A, which is the 1785-ENET sidecar. The term "Full Duplex" is also used for the RS-232 serial Channel 0 port so that might be a source of confusion. Unless your 1785-ENETs are the relatively new Series C hardware, they will have a bulky AUI transceiver and will only be capable of 10 Mb/s Half Duplex Ethernet operation. The Series C hardware is unmistakable because it has a modern RJ45 style Ethernet jack on the front of the module, not an AUI d-shell port. The Series C might even be labeled "10/100" or "100BaseTX". The embedded Web Page will almost certainly also tell you which version of Sidecar you have. I suspect that you're getting enough errors on the port that the controller protocol section of the module is shutting down, but the HTTP server is still ticking along. I know that they are separate processes and I've seen this happen to other types of devices. A trick that I would try next time you catch the controller misbehaving is to use the utility TCPing. This works like the common PING feature, but instead of using the ICMP protocol like PING does, it uses the TCP protocol. Try TCPing using Port 2222 (the old A-B Ethernet protocol) and using Port 44818 (the modern ControlLogix CIP protocol) to see if the 1785-ENET is running those parts of the protocol stack. If you're running Wireshark simultaneously, so much the better.

Share this post


Link to post
Share on other sites
War story time: Last night I found myself armed with an arsenal of taps, switches, cables, and analyzers, attempting to find a problem bedeviling a complex ControlLogix messaging system. A squad of engineers from four companies was huddled on a mezannine above the factory floor, mapping out the complicated physical network and planning our attack. All of us were wearing our flameproof jackets, and safety glasses and foam earplugs to protect us against the hazards of the plant. Riveters were firing away, spray booths clouding the air, cranes whirring by overhead. Standing in front of one of the switch cabinets I took out my earplugs to hear the project manager better and found there was a buzzing, whining warble coming from somewhere in the area. Every time it changed pitch for a few seconds, the error indicators on the HMI flared crimson. I can now reverse my statement that I've never seen a failed N-Tron switch: the power supply was degrading and periodically it would fail enough to reboot either just the fiber ports or the whole switch. We never saw error counters because the switch was rebooting and clearing them. It turns out the problems with the network started a few weeks ago when the area went through a few days of 100+ F temperatures; it was 120+ on the mezannine and who knows how hot inside the enclosure. The switches have been in continuous service for ten years so it's not impossible that some power components just baked themselves. N-Tron remains the far and away leader in reliable Ethernet switches: I've thrown away D-Link gear as a preventative maintenance measure. The moral of the story is not to overlook the physical layer and even the non-endpoint devices. And that sometimes remote troubleshooting can't do the job.

Share this post


Link to post
Share on other sites
You have an opinion on Hirshmann or Moxa?

Share this post


Link to post
Share on other sites
Or Westermo?

Share this post


Link to post
Share on other sites
Hi Sorry I've not got back to you all as I've been on holiday! I've had a look at the mode settings on all our managed switches and they are pretty much all set wrong. Most are set to 10Mbps Full Duplex for the old style 1785-ENET module. I was going to change them, but we have a Rockwell engineer coming on site tomorrow, so I thought it would be better for him to see the network as is. Thanks for all your help and info so far, it's been very imformative! I'll let you know what the Rockwell dude comes up with.

Share this post


Link to post
Share on other sites
I am very interested in what you guys find. I would have done a quick experiment (during the time you say it happens): 1. Make sure your managed switch has IGMP Snooping Enabled. 2. Configure the port connected to the PLC to 10 Mb/s Fixed. 3. Plug in your laptop to another port on the switch (leave it on Auto 10/100 Mb/s). 4. Run the RSView Application on your laptop that has the most tags from this PLC. 5. Run RsLogix 5 and get online. 6. Disconnect uplink on switch. 7. Check if the problem still exists. Easier to troubleshoot locally. If it does its possible that there is a hardware problem with the ethenet adapter or the switch. If it doesn't then its most likely you have a problem on your network (connections, bandwidth, traffic). *For all my wired Ethernet, I try to use only N-Tron switches. At least a 508TX-A. *For all my wireless Ethernet nothing beats the SCALANCE Industrial Wireless APs and Clients from Siemens!! Nothing! (especially if you have wireless carts or vehicles that move from room to room - roaming)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0