d0ubleth1nk

L35E CPU Ethernet Fail

14 posts in this topic

I have a CompactLogix L35E where the ethernet portion of the CPU dies every 2 months. The CPU still runs and the plant is fine, but the HMI and OIT are blind until I connect serially to the RPU and reset just the ethernet. The "NS" light blinks continually during the error and which means: "The port has an IP address, but no CIP connections are established." Why is the ethernet portion of the card hanging? Everything is hardwired and it's just the single RPU so no ethernet messaging or anything. Thanks.

Share this post


Link to post
Share on other sites
This is a shot in the dark but are the IP address / Subnet Mask / Gateway address / and check boxes correct under the Port Configuration tab of the Ethernet pprt configuration window?

Share this post


Link to post
Share on other sites
is bootp still enabled in the config?

Share this post


Link to post
Share on other sites
I'm pretty sure the configuration of the ethernet port is okay as comms is fine for 2 months. Bootp has been disabled the whole time. The only thing sticking in my mind is the RPI under the "Connection" tab is static at 0 ms. Even offline I can't modify it. There's also an RPI under the backplane that I have set at 100 ms. From what I've read the RPI has to be something greater than 2 ms, but if both of the RPI's are the same then I'm fine.

Share this post


Link to post
Share on other sites
What are the HMI and OIT devices ? Do they give you any error messages that might be of use ? What kind of switch is the CompactLogix connected to ? When this occurs, have you done any network connectivity troubleshooting, like using PING or trying to browse the CompactLogix with a web browser or RSLinx Classic ? What exactly do you do to "reset" the Ethernet daughtercard in RSLogix 5000 ? If there are no I/O devices configured in the CompactLogix I/O tree under the Ethernet module, then there don't need to be any Requested Packet Interval (RPI) values configured for the Ethernet module itself, so that's not the issue.

Share this post


Link to post
Share on other sites
The HMI is Wonderware and the OIT is a Panelview. Both give general Communication failure alarms on timeout. No error codes that describe what's happening at the RPU level. I always spec a Phoenix Contact switch, that has never given me trouble in the past. When the CPU ethernet hangs, the network is still alive and well where I can connect my laptop and ping everything except the CPU. I cannot connect to the CPU through the ethernet port, but I can still go online serially. RSLinx has the CIP diagnostic feature that mentions the CPU is not accepting CIP connections. To reset the ethernet portion there's a reset button under the ethernet properties and "Module Info" tab. In using the web browser after restting the ethernet card, the stats page tells me my "CIP Msg Connection Limit" is 32 and "Max Msg Connections Observed" is 8. I'm not a network expert but all the stats appear good and nothing jumps out as being problematic.

Share this post


Link to post
Share on other sites
Let's investigate to see if there is a problem with the Wonderware driver or the PanelView driver. It's possible that there are communication failures that you're not seeing, whether caused naturally (wiring, noise) or at the software level. If either the Wonderware or the PanelView clients fail to properly shut down the CIP connections to the controller when there is a communication failure, you could be running out of CIP connections. Think of CIP connections as being like phone calls. The call is placed by dialing a number, ringing, and picking up (the TCP connection), but the conversation that takes place on the phone is separate (the CIP connection). The Wonderware and PanelView stations will only open one TCP/IP connection, but they'll run multiple CIP connections. A PanelView can run up to five CIP connections, for data displays, alarming, trending, and Global Connections. I don't know how many Connections a Wonderware DASCIP driver can open. Is there any editing going on, while this system is operating ? What else is connected to this network; is it connected to a large LAN ? I wonder if the CompactLogix ARP table could possibly be full. The fact that you can't PING the device strongly suggests a complete network failure, though. If this was a CIP problem, the HTTP and PING services would still work. Keep checking with the web pages while the system is running to see if there is an increase in CIP messaging connections being generated by one or more of the devices.

Share this post


Link to post
Share on other sites
I can think of two things to check. 1) Have you tried a new or different Enet switch? I had a similiar problem (not exactly as yours) a few months ago. The local techs at the site would just cycle power to the L35E PLC rack when the ethernet comms locked up. I installed a new Enet switch and it's been running without any comm problems. 2) You might contact your local AB rep to verify if the firmware version you are using in your L35E has any issues. Last year we had a ControlLogix processor dump it's program every 3 months. After the 3rd time I contacted my local AB rep and gave him the model, serial, and firmware in the processor. Turned out that the firmware we were using had a "glitch" in it. ( don't you love the techincal word "glitch" ). I uprade the firmware and it's been running non stop ever since.

Share this post


Link to post
Share on other sites
I definetly could not ping the CPU during the failure so I suppose it's not a CIP problem. Also the stats page says that I've never maxed out CIP connections as there's a total of 8 from the Panelview and DASCIP. This isn't a man'd plant so the failures happen during normal operations and no editing. The network is comprised of the CPU, HMI, OIT and a SLC505 for a vendor grinder system at headworks. Come to think of it, comms to the SLC is not trustworthy as it goes in and out. The contractor is trying to get the electrical sub to pull a new CAT 6 cable. There's been much bigger issues to deal with, but I should finally get around to looking at the SLC comm fail and see if that is somehow tripping the Compactlogix. We recently upgraded the firmware as Rockwell always friggen suggests firmware, but in my experience it rarely does anything. Thanks for the suggestions!

Share this post


Link to post
Share on other sites
Dear all, I have a similar problem as well. The connection between Compactlogix L32E and Intouch Wonderware is fine for 2 weeks. But it become no connection after that 2 weeks time. CIP Msg Connection Limit is 32. Max Msg Connection Observed 31. Is that csued by the Maximim Msg connection observed very near to the limit? Thanks.

Share this post


Link to post
Share on other sites
Yes, the CIP connection limit is very likely a problem. Do you have a lot of I/O on Ethernet connected to this controller, or more than one Wonderware session, or other HMI/SCADA software ? The diagnostic webpages on the 1769-L35E will help you figure out all the devices the controller is communicating with.

Share this post


Link to post
Share on other sites
Hello, I am having this exact same problem. to add some info. we are ably to restore communications by simply unplugging the ethernet cable and plugging it back in. No reboot needed. Also, our CIP connections are good. But after this problem we do see some FCS errors counted in the port statistics. Regards,

Share this post


Link to post
Share on other sites
On 1/4/2011 at 0:33 PM, d0ubleth1nk said:

The HMI is Wonderware and the OIT is a Panelview. Both give general Communication failure alarms on timeout. No error codes that describe what's happening at the RPU level. I always spec a Phoenix Contact switch, that has never given me trouble in the past. When the CPU ethernet hangs, the network is still alive and well where I can connect my laptop and ping everything except the CPU. I cannot connect to the CPU through the ethernet port, but I can still go online serially. RSLinx has the CIP diagnostic feature that mentions the CPU is not accepting CIP connections. To reset the ethernet portion there's a reset button under the ethernet properties and "Module Info" tab. In using the web browser after restting the ethernet card, the stats page tells me my "CIP Msg Connection Limit" is 32 and "Max Msg Connections Observed" is 8. I'm not a network expert but all the stats appear good and nothing jumps out as being problematic.

Our company is a system integrator and we get involved at many different businesses. This issue has come up twice for us, but both times it has been at the same company, on the same control system, but with different L35 processors installed. Resetting the Ethernet module seems to fix the issue temporarily so thank you for the note on that  d0ubleth1nk!  

 

Has anybody resolved the cause of this yet?

Share this post


Link to post
Share on other sites

So today we fixed this issue. We had the ms (module status) and ns (network status) solid red and flashing red light on our L32e processors. The fix was to update the firmware. The two processors had version 16 and 17 installed we updated to version 20 and that fixed our issue! We found that re-seating or power cycling the processor would get us the green lights for about 20 min then need to be done again but after the version update it has been working great. Hope this helps someone because we had to do some troubleshooting to find the fix. Thanks for all the information it helped us out!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now