Sign in to follow this  
Followers 0
Pierre

PLC5-40C with ENET sidecar MSG problems

8 posts in this topic

I have 3 PLCs. No.1, 2 and 3. I am having some errors between No.2 and the two others. Using the MSG block to READ from the other 2, they all do the same thing... BUT... I always have errors when a READ is tried from 1 to 2 AND also from 3 to 2. All other MSG functions are OK. So 1 reads 3, 2 reads 3, 2 reads 1 and 3 read 1. The error code I get from the message element is Hex 18 - connection was broken. The connections are through the ENET module. I am now experiencing this for the last 36 hours. Sometimes, the problem goes away for up to 10 days. But now its back. I monitor the seconds register in all 3 PLC from the RSView Tag function and I never miss a second... even when I get these errors. Cables have been tested. There is 2 cables at each PLC and the give us same results. Cisco switch has been tested and replaced... just in case. Rockwell support people have spent a full week here with no results. They suggest using one network specialist... from there team. It is starting to be VERY expansive and we are trying to search for the problem a little further before we have them come in again... This proble has happened in the past, once every few months. Now its there since 2 days ago. Any ideas?

Share this post


Link to post
Share on other sites
If I were in Tech Support, I'd be asking for your programs (to examine how many MSGs and the diagnostic files inside the controller) and for the firmware revisions of all the controllers. Then I'd create a Mirror Port on the Cisco switch and monitor all the traffic from the port connected to Controller 2. Wireshark software is the open-source and free standard, but professionals in industrial software use Frontline Test Equipment's NetDecoder. The "connection broken" message means what it says; the TCP connection has been broken between the controller originating the message and the controller receiving it. What you need to do is determine at what stage that TCP connection broke, and whether it broke because it timed out or if it was closed by one of the participants or closed off by a switch or router.

Share this post


Link to post
Share on other sites
Let's make sure we know which messages are working and which are failing. Your description is: The first line I'm going to call Messages A and B The second line I'm going to call Messages C,D,E,F. Does this diagram match what you're describing ?

Share this post


Link to post
Share on other sites
Wow, that was fast :) Thanks. First I must add that last summer we upgraded PLC no.2 system to Cnet 1.5 AND we upgraded the other two to 100MB sidecars. Now we have 3 PLCs at 100MB and one is at Cnet15. The work of upgrade was done by Rockwell people. We replaced the ACNs (22 nodes). Last Friday, PLC no.1 was disappearing from the network... to a point where we could no longer access it. We needed to use the serial link with AIS software to re-configure it and download its program. When it was not communicating, it snooped the Ethernet with Wireshark and kept the file. We had to power down the no1 PLC and lower the com speed to 10 MB. Now it is working but at 10MB only. PLC1 and 3 only have 2 MSG blocks. Reading files from the other 2 PLCs. PLC 2 has 8 of those. 2 to read the 2 other PLCs and 6 to read/write to a SLC5/03 over DH+. (all of these give no errors) A will go and record the actual network and get back.

Share this post


Link to post
Share on other sites
OK, I remember your ControlNet upgrade last year. The changing of the Ethernet speed sounds very suspicious. What made you conclude that changing the data rate to 10 Mb/s was necessary ? When I have devices that support Auto-Negotiate, my rule of thumb is to let them auto-negotiate. The combination of an Auto-Negotiating device with one that is hard-set for a particular speed and duplex is the source of most problems that I see with port configuration. Your Cisco switch should provide some statistics for the port and status about its speed and duplex.

Share this post


Link to post
Share on other sites
I was hired to revamp one section of this plant. This section is mostly controlled by PLC No.2. As I was working on it I saw that the rest of the plant was having big problems with slow Ethernet communication. Imagine this: They where controlling the position of one piece of equipment by the touch of a button on one RSView screen. When the com started to slow down (after they loaded there trending software with tons of data) they could hardly click at the good moment... So they had a recommendation from Rockwell to upgrade there systems to CNet15 and also 100MB on the sidecars. Of course I only got to upgrade one PLC to but we included the sidecar upgrades. Now the IT guru tells me the network is 17% loaded only... so it should not here be an issue. The setup is like this: 2 x PLC-5/40C (each around 14 ACN FlexI/O nodes) 2 x ENET 100MB module 1 x PLC-5/40C15 1 x ENET 100MB module 4 Main control room PC with dual screens (RSView + RSTrend) 1 Secondairy control room PC with dual screens (RSView + RSTrend) 3 Manager PC (RSView + RSTrend) 1 DEV PC (RSView + RSTrend + RSLogix5) 1 Data PC ( (RSView + RSTrend)) 2 Gateway Server PC (RSLinx Gateway #1 & #2) After I left the MrPLC site, I went to the PLC No.1 and connected to the dumb hub (10MB) that is there. I saw that the PLC was having com errors (with Wireshark) so I went to the main control room and called there IT Guru. Through his dedicated link he connected to the routers and had the PLC No.1 port mirrored to one that I connected my laptop. Saw again the same errors (BAD TCP to 192.168.5.109) All the errors where to that same machine (PC) We did the same thing to the other two PLC and saw some errors, and some where toward that same machine but only a few. We checked what was that machine and it was the one carrying the TREND data. Smart has he is he said "Why don't you disconnect the TREND machine... we won't need it for tonight." Smart has I am I did. We completely lost communication to and between all PLCs! The plant was running blind. I could not connect to the PLC No.1 and I had to power it down. Took us 90 minutes to start the plant back up. Some day your the windsheild, some day you the bug. After this the secondairy control room PC could not connect to PLCs No.2 and 3. And this is where we stand. I will add more details after a good hot shower. I need it.

Share this post


Link to post
Share on other sites
The PC which don't connect to the PLC can browse the web page inside the ENET module but RSLinx just wont connect. Here are the 3 file morrored before the big failure. PLC 1 = 192.168.5.5 PLC 2 = 192.168.5.6 PLC 2 = 192.168.5.7 Wireshark_3_PLC.zip

Share this post


Link to post
Share on other sites
This is what Wireshark gives me as statistics:

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0