Ethernet Troubleshooting

[BobLfoot 301]

I am having an issue with a ControlLogix PLC installation which I am relatively sure will be a network issue, but I don't want to overlook something.

The system was originally a 1756-ENBT\A with a 1756-L61 CPU which we upgraded today with a 1756-EN2T\C.

We are chasing a problem where the FCS errors are occurring at the rate of 1-8 per minute.  The 1756-EN2T has been running for 4 hours (240 minutes) and accumulated 685 FCS. {See EthernetStatistics.png}  Now my IT counterpart swore he saw no issue at his switch ports and the issue should have been the 1756-ENBT\A.  Todays results would dispute that.

Now looking at connection counts I noticed that both CIP and TCP connections cycle from a low value 7/8 respectively to a high value 13/25 and then back to the low value count.  Not sure if there is a way short of wireshark at the switch to ferret out the source of these connections.

I did notice that the TCP Connections list shows three connections from the same IP address {A Scada HMi Node} having the same serial number and very low uptimes.  And they never seem to age much before they close and are reopened.  I'm wondering if I have a configuration error on some SCADA hooks that I need to find and address?

I'm open to any suggestions as to where to look next.  Has anyone had issues like this between their PLC and their SCADA.