Sign in to follow this  
Followers 0
Amish

Stratix 8000 and IGMP issue

8 posts in this topic

Hi all, I have an issue with a remote site that I just set up. It includes three Stratix 8000 switches - one is talking to two 1734 AENT remote I/O and the switch in the middle. The switch in the middle talks to the other two switches and another 1734 AENT, and the switch on the end talks to the middle switch, two SMC Flex soft starts, a PVP 1250, a CompactLogix L32E, and a radio link. The other end of the radio link ties into another Stratix 8000 along with a SLC 5/05 and a fiber link to another Stratix, which ties in to the other end of our plant loop, another 5/05, and our OPC server. I can figure out why, but what seems like every morning, the IGMP snooping will stop working and multicast packets from the I/O and soft starts at the remote site will be transmitted over the radio link, dropping out the link, as well as everything else the server is reading, once in a while. Anybody have an issue like this? Does the location of the querier in the network matter? Firmware issue? I was sent a couple links to knowledgebase articles by AB, but of course our IT department didn't renew our tech connect contract. Thanks for any help.

Share this post


Link to post
Share on other sites
Can you post a diagram of your network setup? Is your "switch in the middle" a managed switch with IGMP snooping turned on? If I'm reading your description correctly then the muli-cast traffic will reach the radio link.

Share this post


Link to post
Share on other sites
The switch in the middle does have IGMP snooping enabled. It is also the oldest F/W revision of the stratix 8000 (44). I've attached the network layout of the area. I managed to capture the radio feed this morning as the IGMP "stopped working". I can't see anything really obvious (at least to me, not saying much), but once the IGMP stops working, there is a substantial increase in requests to join a multicast group (Several different groups) coming from the compact logix attached to the switch at the remote site (5.0.5.71 on the network layout). Also, I start to get IGMP querys from both the switch at the remote site (5.0.5.171) which is what I intended, but also from the switch at the other end of the radio link (5.0.5.175). Another thing I found out was that the switch at the remote site MCC (5.0.5.171) is the latest firmware, which on the AB website for stratix 8000 firmware updates, it doesn't list it as an available/applicable update for Logix 5000 v17.x, but I don't know that that matters.

Share this post


Link to post
Share on other sites
We have recently learned a lot about IGMP snooping, much of it the hard way. There are different versions of IGMP snooping, and you need them all to match the querier version. I believe certain versions may allow multiple snoopers, but our Hirschmann switches did not. And, we have several firmware levels of our RS series switches, which have been 95% hardware reliable, installed over many years. I think two of them turned off their "IGMP enabled bits" for all the ports and dropped our network performance, luckily the impact was minimal, one particularly sensitive FTViewSE Standalone HMI always slows down first, but still functions usually. So, I started wireshark on my local connection on the controls side of our LAN, and saw all sorts of traffic coming in from an I/O module behind a managed port! I go to that switch using Chrome and it's assigned IP address, turn on IGMP, save it, save an offline copy of the config, and it seemed fine. Wireshark immediately calmed down, and flex i/o traffic all over our network ceased. Four days later, when I was off, Mike said he found two other switches with IGMP turned off, and it caused his PC to slow down due to Point I/O modules on the other switch. I think he repeated the same steps, but may have upgraded the firmware in the oldest one. I think it would not retain that setting following a power cycle, and we had a power loss on those UPS backed switches at some point before the problem was reported during a planend outage on a weekend. The 2nd switch, I believe he replaced since it was an older hardware level that was not easy to update. I know your switches are cisco/rockwell and not hirschmann, but the possibility may still exist for something similar to occur. Find out which version of IGMP you have configured in all your switches, and make sure they didn't somehow magically disable IGMP snooping like in our case. Our first experience setting up IGMP, we learned that it is not wise to use version 1.0 and version 2.0 settings among switches on your network. I don't recall the details, but we had no problems until this spring with the one switch losing it's settings. Edited by OkiePC

Share this post


Link to post
Share on other sites
So I'm still looking into the issue, but I did notice one thing yesterday that must've slipped by me - the RPIs for two of the Point I/O and the soft starts were all set to default (20 and 100 ms respectively). I set those up to 400ms for the I/O and 800 ms for the soft starts. It seems to have abated the issue, but multicast traffic will still make it through the radio every hour or so. I don't know why doing that would make a difference though. I put the system together as best as I could in Rockwell's Ethernet/IP capacity tool, and the only limitation that came close was the number of CIP connections, but nothing was exceeded, even setting the RPIs to default. I'm not sure on the version of IGMP in the switches. I think they are all v2, but I will have to check.

Share this post


Link to post
Share on other sites
This may be overly simple, but wouldn't your first step be to restrict IGMP packets from using the port of switch 171 which has the radio link connected to it. Then tackle why and where the IGMP relay from Logix Processor to End hardware is dropping out. You switch with the PVP plugged into is suspect to me. CAVEAT - I've not personally setup and used the stratix switches although we have tons of them in our plant. Engineers handle the switches. We maitnenance folk are a "security risk" after all.

Share this post


Link to post
Share on other sites
IGMP is only required to limit multicast traffic being broadcast to devices that can't handle the extra traffic. Are you using any ListenOnly type connections? If not, can all your devices support Unicast? If so, then a solution is to replace all of the multicast input types to point-to-point (which is unicast UDP traffic). IGMP would no longer be required. This assumes that the various hardware/software all support unicast, which may not be true especially with older firmware/software revisions. I try and do the following things: 1. Avoid Listen Only connections, which require Multicast 2. Use point-to-point type whenever possible 3. Engineer the system so that devices can survive the presence of the Multicast traffic that must exist 4. Disable IGMP I realize in the presence of Wireless, it's a little tough to just live with Multicast traffic if it must be there due to the limited bandwidth. But some ways to deal with this: 1. Route to the wireless network as the default TTL is 1 for EIP Multicast packets - so they won't get through the router to the Wireless net. This could add significant complexity and expense. 2. Make sure that no switch upstream of the wireless net issues IGMP queries - my experience is more with Hirschmann and Schneider Connexium switches, and they have a setting to send Multicast packets to query ports. So if the switch gets a query from the wireless side of the net, it could send all of it's multicast traffic in that direction. I think others have mentioned this already.

Share this post


Link to post
Share on other sites
Here's an update: Instead of fighting the IGMP snooping, I decided to do what I originally intended (but went against in order to save money by using stuff I had on hand) and replaced the L32E with a L43 and two 1768-ENBT cards. I put all the soft starts, I/O, HMI and switches on a separate local network, and used the other card to talk to the radio link over the SCADA network. This definitely stopped the Ethernet/IP multicast traffic, but I still found essentially the same issues with the SCADA network. Now that the Ethernet/IP traffic was gone, I could run Wireshark from my office and clearly see the points drop off the SCADA program as certain spanning tree packets showed up on the capture. What the IT guy found was his spanning tree was periodically dropping out and putting the network in a loop. He changed out one of his switches, and I don't know if this played any role, but the Stratix comes with spanning tree enabled by default, so I disabled spanning tree on the two Stratix switches I still have on the SCADA network as well. He said when he first put in the Cisco catalyst switches for the SCADA network, he had several issues when spanning tree was enabled on all the switches. So, even though the two Stratix switches I have installed are not in a place where they could make or break a loop in the network, I disabled spanning tree just to be safe. Before, you could see spanning-tree packets coming from these switches, so they must have been up to something. So far, there have been no network dropouts :) For those curious on how to disable all spanning tree activity on a Stratix 8000: The RSTP checkbox, at least on my Stratix, is greyed-out on the web interface, so go in through the command line interface. "no spanning-tree vlanx" will disable STP on whichever vlan you are concerned with. "no spanning-tree mode" will disable MST and RSTP on the switch, as well as allow the RSTP checkbox to be selected on the web interface.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0