Strange DeviceNet behavior - Bus Off error

[phuz 2]

Just got back from visiting a customer on a trouble call. The system is a ControlLogix L50 running version 7. Yes, I did say SEVEN. :D Apparently an upgrade was attempted several years back to bring it to version 12 or 13 and it did not go so well, so the original processor was left in place. The rack has a CNB in slot 1, DNB/A in slot 2, DNB/D in slot 3, and the rest are IO modules. Both DNB modules have 24vdc powered from the same power supply (24vdc @ 40A). The customer complained of the DeviceNet on slot 3 (DNB/D) dropping out, and when it happens the module states "Bus Off Detected." During this, when trying to scan it with Networx, none of the nodes are discovered. Under the input array of the DNB module, the fault bit is not set, but we have DeviceFailure and CommFailure. In order to clear the problem, the customer discovered that simply removing the DeviceNet plug from the front of the module and reattaching it is sufficient. No power cycling needed to the rack and no need to pull and re-seat the module. I found this strange. After this, everything comes back online normal and they continue running until the next random failure. The process runs its normal routine over and over throughout this time and they have no way of reproducing it. All wiring terminations were supposedly checked. I asked to check the voltage of the power supply and found it to be 23.35. A little low, but within tolerance. For the sake of elimination, we adjusted it to 24.00. I figured if there was a voltage issue, the other DNB module would be experiencing issues as well as the 24vdc IO. One other thing I noticed is when I tried to toggle the Reset bit in the DNB output command register, it faulted the processor. Clearing the fault and then toggling the RUN bit actually was sufficient to get the DNB module back online. Of course the customer is aware of the incompatibilities of the version 7 Logix and the DNB/D and will make another attempt at an upgrade in the near future. This system runs fine for months and months and then has its bad days, like today, where it will fault out several times forcing them to remove the DNB plug and reattach it. I am heading back there in the morning to be on standby. Any thoughts between now and then are appreciate.

[Ken Roach 194]

Some of what you're describing is normal. The Bus Off fault means that the 1756-DNB isn't getting a readable signal from the DeviceNet network. It's usually because of a short or open circuit on the blue or white wires, or a large amount of noise on the network, or a device transmitting incorrectly on the network. Bus Off does require a power cycle of the DeviceNet power into a device that has that fault. In most cases, all the devices on the network sense the bad signal at the same time and they all go "bus off", which requires you to power cycle the DeviceNet power overall. It's less common, but still possible, for the 1756-DNB to detect Bus Off when the other devices don't, and to only require the DeviceNet power to be removed from the 1756-DNB's front port. Bus Off isn't a failure of any of the subsystems of the 1756-DNB, so it makes sense that the module's Fault bit isn't true. The Device Failure table is the most common place to detect the failure of connections on the network. Setting the "RESET" bit in the command word actually performs a warm re-boot of the module. If the PLC is configured to fault when the connection between the 1756-L55 and the 1756-DNB fails during runtime, then the PLC will fault when you do that. I'm concerned that the system uses a single very large DC power supply for several DeviceNet networks. This is contrary to several of the basic design principles of DeviceNet networks. You shouldn't run more than one DeviceNet logical network from the same power supply, because noise on the DC common could mess up both networks. DeviceNet networks should have a dedicated power supply that provides power only to the network ports. That's the point of having +24V power on the network cable; all the transceivers are isolated from the bulk power circuitry and the I/O devices. You should never run the I/O circuitry, especially the Output circuitry, with the same power supply that runs the DeviceNet network. Obviously the 40 ampere supply is running one or more very large loads as well as the network. The heaviest network cables for DeviceNet are rated for 8 A. Hopefully the system has fusing to protect those cables from the 40 ampere power supply and isn't attempting to draw significant amounts of that power through the same wiring as the DeviceNet trunk.

[Ken Roach 194]

I very strongly recommend you pursue troubleshooting of the physical media. Because the other 1756-DNB doesn't fault, and because other slave devices don't appear to fault, I'll bet the problem is in the wiring between the troublesome 1756-DNB and the rest of the network. "We checked the wiring" often means "we stared at it, really hard, and didn't see any broken wires". At the very least, every screw terminal between the DNB and the rest of the network should be taken apart and re-assembled. The pro-level tools for this sort of thing are the SST/Woodhead "DeviceNet Netmeter" and/or a digital storage oscilloscope. I was discussing an unusual DeviceNet failure with a colleague of mine on another Forum last week and he reported that eventually he had found a couple of photoeyes whose cordsets had been worn through and were shorting for a millisecond when a bit of material would jostle them. That interruption on the CAN signal was enough to cause a timeout of an I/O connection but not a Bus Off fault. Firmware doesn't change, and in my experience modules very seldom degrade. But connectors and cables start degrading as soon as they're put in service.

[phuz 2]

Thanks Ken. They did inform me that they actually went around and tightened connections, but there are about 30 nodes on this particular DNet and I don't know to what extent it was done. I was very surprised that the IO and DNet were being fed from the same 24vdc, and cautioned it and maybe I can at least have them segregate that.

[phuz 2]

Connections are good. We've had it trip three times this morning. The strange phenomenon, which falls into Ken's "less common" category, is that some of the nodes go RED when the Bus Off occurs and some remain GREEN in their fat, dumb, and happy state. Next step is to put a scope on the CANL and CANH terminals and check for noise.

[phuz 2]

Here is a quick video of the scope. The first part of the video shows what I would call "normal state" of the CAN_H activity when the system is idle (even though it is still not the prettiest). Toward the end, you can see it get very ugly and this is occurs whenever certain equipment would run. I did find out today that there are several locations where the DNet cable runs parallel to 480 lines which are powering several 7hp motors. Today we did add a power supply strictly for this DNet, but the OEM had still tapped off some of the DNet's 24vdc power in the field to power some relays, etc. While we did manage to remove the majority of it, there is still some IO powered by it. After that modification, we had no Bus Off issues over a 3 1/2 hour period before they shut down for the day. Coincidence or success, it's too early to tell. I also learned that this issue occurs more in the morning than any other time and I am wondering if maybe the motors are drawing more current because of cold conditions and therefore inducing more interference on the DNet. However, based on the scope video below, would you consider this interference, alone, to cause a Bus Off? IMG_3068.MOV Edited 20 Feb 2014 by phuz

[phuz 2]

Update: The system started up at normal time today and has 6 1/2 hours of clean running without a Bus Off problem. Separating the power supply might have helped, but time will tell. There is still a lot of cleanup to do with the field IO powered off the DNet power.

[Ken Roach 194]

I can't say from that view of the waveform if it's error-causing noise or not. You'd have to get down to a static capture at a higher resolution to look at the edges of the waveform, and even then you'd only be looking at part of a packet. That's why things like the NetMeter are so great; they have a CAN transceiver chipset inside, so they can tell for sure if a particular noisy waveform is balanced out by the differential circuitry of the transceiver, or if it counted as an error frame. I use a Fluke 125 ScopeMeter in conjunction with a NetMeter when I'm getting down and dirty with DeviceNet. 480V output from a VFD is a notorious way for noise to get into DeviceNet networks via the power conductors. I remember doing a retrofit MCC when Yaskawa first came out with their DeviceNet modules for AC drives, and the installer put the DeviceNet power supply in a spare bucket on the far left side of the MCC, then ran ordinary untwisted blue wires all the way through the upper raceway over to the PLC cabinet on the right side. As soon as the drives started running, the network started faulting on Bus Off errors. They refused to let me move the power supply, so I just connected it to the nearest drive's DeviceNet connector instead of to the PLC scanner module. The twisted pair inside the DeviceNet cable is a lot more noise-immune than wires that were just laid in a raceway, and that solved almost all of the network faults we had in that system.

[phuz 2]

It's amazing to see the shortcuts that some OEMs take to save a buck.