ML 1200 and radio modem problem

[mjrx 1]

I have 4 ML 1200's on a radio network. I am using Ch 0 serial, running half dupex slave 9600 N81. I have tried No Handshaking and half duplex modem, and about every timer setting in there. What happens is all is well for quite some time and then the serial port locks up with no errors, and the PLC continues to do its thing running the local process. Sometimes it will restart and start running again on its own, but cycling the power to the PLC will always get it going again. Its like a buffer is getting full in the ML1200 or something. At first we thought it was the radios, but have definitely found it is the PLC. We are not transmitting much data, about 24 bits and a dozen or so words, and each radio gets polled every 15 seconds. This one is kickin' my butt. Anyone have an idea?

[Michael Lloyd 85]

How are you addressing each ML1400? You can't put them all on the same address.

[mjrx 1]

[How are you addressing each ML1400? You can't put them all on the same address.

[Ken Roach 194]

Don't just shotgun serial settings at the thing. If it works "for a while", the problem is very unlikely to be the serial settings. What's the DF1 master ? Are you really using DF1 Half Duplex or are you using DF1 Radio Modem protocol ? Are you using Polling or Message-based master communications ? Are there MSG instructions inside the slave MicroLogix controllers ? Are you running diagnostic files inside the slave controllers for the serial port ? What model of radio are you using ? Does the radio have an option for hardware handshaking ? The two circumstances I've seen in which a controller "locks up" and requires a power cycle are when too many MSG instructions are queued up without being complete or errored, or when a very large number of serial errors have occurred and the controller eventually stops responding to serial requests.

[mjrx 1]

Yes I agree that throwing serial settings at it is the wrong approach. It can work very well for awhile, then crash (can run OK for up to a few days, other times just hours). The DF1 master is NI Lookout SCADA software, set as half duplex master. Each PLC is indeed set for half duplex slave, addressed 1 - 4. I didn't try DF1 radio modem because I don't think my SCADA supports it, it only has full and half dupe DF1 as options. We are using Calamp Gaurdian licensed UHF radios, 450 mHz. They are currently operating in DOX mode, but I do have hardware handshaking available. I am not using any MSG instructions. Th SCADA sequentially polls a PLC every 3.75 seconds, so data is updated from each PLC every 15 seconds. Very little data - just the I/O bits and read/write of a dozen N7 regs and a few B bits. The entire polling cycle appears to complete in less than 1 second, so there is a good pause between polls. It just feels like some kind of buffering issue in the PLC, but I can't find any AB info or docs to support this. Thanks for your help.

[Ken Roach 194]

Rule #1 with radio modems is that they ALWAYS drop and corrupt data. They might not do it often, but they do it, and that's why devices on both ends need to be able to handle it. The MicroLogix 1200 stores Channel 0 serial diagnostics in the CS0:xx function file; if you right-click on the diagnostic counters in the Channel Status you'll see which ones mean what. I would copy CS0:10 through CS0:18 to an Integer array and make it part of the data you're reading with the DF1 master. That will allow you to remotely determine if there are intermittent or recoverable errors that are happening in between the unrecoverable port issues. Also try disabling Duplicate Packet Detect. I don't know anything about NI's implementation of DF1, but I've seen devices that literally re-transmit every byte the same when they perform a re-try of a message, rather than properly incrementing the Transaction Sequence Word. A MicroLogix that has Duplicate Packet Detect enabled and gets multiple transmissions of the exact same packet will ignore all the subsequent ones. In such a case all it would take is one bad bit in the transmission from the MicroLogix to the master to cause the MicroLogix to never reply again.

[mjrx 1]

I have retrieved the CS file info. I have disabled Duplicate Packet in the PLC, no help. Below is the CS0 file immediaetly after comm is re-established. CS0:10=27043 CS0:11=27056 CS0:12=8 CS0:13=123 CS0:14=2 CS0:15=27485 CS0:16=26 CS0:17=361 CS0:18=0 I have the explanation of the registers and 10 - 12 are simple enough, but I'm unsure about CS0:17 "No Buffer Space", I'm not sure what the value 361 means and CS0:13 "Message Packets Retried" does that mean 123 packets were retried to send from the AB end? Is there a more setailed explanation of this CS0 file somewhere? And I assume CS0:18 is reading 0 because I have duplicate packet detect disabled? Thanks again.

[Ken Roach 194]

I think you're right on about the "Message Packets Retried". Master: "Tell me the the value of N10:4, length 3" Slave: "That's six bytes of data, and the values are xxx, yyy, zzz." Master: "Th-nk you" A timeout period goes by. Slave: "That's six bytes of data, and the values are xxx, yyy, zzz." (to self: increment Retry counter) Master: "Thank you" I'm less sure about the others. Take a look to see if the classic A-B "SCADA Application Guide" has been updated to discuss the MicroLogix CS0 files.

[mjrx 1]

Very good explanation. I have looked at the A-B SCADA guide and CS0:17 has me concerned: The manual states: The number of times the processor could not receive a message because it did not have available memory My value for this was 361. How do I manage this part of "memory"??I am wondering if this is the buffer overfill concept I am sensing that may be the issue? ? ? The number of times the processor could not receive a message because it did not have available memory

[Ken Roach 194]

I wish I knew exactly what that "available memory" description meant. The MicroLogix 1200 controller can answer 20 polls a second without breaking a sweat, and they don't dynamically allocate memory the way ControlLogix does. Are there any local HMI devices connected to the controller (this would be the 1200R model) ? How much unused memory do the controllers have ? From the context of being a setting for the MicroLogix serial port, I would think the "memory" is the incoming data buffer. Imagine a packet with a malformed end, so the controller doesn't see the ETX and the CRC bytes and just waits interminably for the next packet, but a new packet arrives without the buffer "timing out". If it were my system, I'd start working this question up the ladder at RA Technical Support. They might have to go to Commercial Engineering for answers, but they'll find some.

[mjrx 1]

There is one HMI on one of the nodes, the other three have none. The one with the HMI is not the worst offender. All of them crap out occasionally. I have used this radio scheme many times with no problems, with PLC' from Mitsubishi, Siemens, and A-B SLC500's. The radio's seem rock solid, RSSI of around -70 dBM, all four. If the buffer explanation holds true you would think there would be a way to flush that buffer, but I find nothing (user available anyway) to do this. There doen't seem to be alot of options for managing this port/protocol other than the usual RSLogix setup. I would think A-B would handle errors a little better. . .

[Ken Roach 194]

For MicroLogix 1200 controllers with FRN7 or higher, the ASCII Clear Buffers (ACL) instruction will also clear the DF1 buffers. Have you been able to monitor changes in the error counters ? We know the total number of errors over time but we don't know how fast they're accumulating. I've never tried writing to these registers in ladder logic; you might be able to clear them by using the Copy Word (CPW) instruction. CS0:10=27043 Total Message Packets Sent CS0:11=27056 Total Message Packets Received CS0:12=8 Undelivered Message Packets CS0:13=123 Message Packets Retried CS0:14=2 NAK Packets Received CS0:15=27485 Polls Received CS0:16=26 Bad Message Packets Received CS0:17=361 No Buffer Space CS0:18=0 Duplicate Message Packets Received "Undelivered Message Packets" means the MicroLogix replied to a data request but never got an ACK packet from the Master. "NAK Packets Received" means the MicroLogix replied to a data request and that data was corrupted in transit, causing the Master to send a NAK reply. "Bad Message Packets Received" means the data values didn't match the Checksum. All of those counters indicate corruption of data during its transit of the radio network. This doesn't mean there's anything wrong with the radios, just that normal error checking and error handling is happening. I would change the "Message Retries" setting in the MicroLogix Channel 0 setting to from 3 to 0. I don't know how the NI driver handles timing, so if it doesn't wait for re-tries, it might just go on its merry way polling the next controller without waiting for a MicroLogix that's trying to get an ACK or NAK after sending data to the master. I'd like to have a detailed explanation of what circumstances lead to the "No Buffer Space" or "No Memory Available" counter incrementing. If you do get a reply on this from RA Technical Support, please share it. Edited 22 Feb 2012 by Ken Roach

[mjrx 1]

Setting message retries from 3 to 0 was an improvement. I would like to try the DF1 Radio Modem protocol, but what master protocol would you use with this? If I were using RSLinx, would I just use the DF1 Polling Master? NI Lookout only has DF1 Full and half dupe as protocol selections, so it may not work with NI - I would believe it (NI) would still send out the full packet structure of DF1 half dupe and the slave set as radio modem may not recognize it (???) Is the DF1 radio modem just another type of half duplex mode?

[Mickey 71]

See pdf below for info on the DF1 Radio Modem Protocol. Once the port is configured for that you can still go online with the processor using the RS232- DF1 driver, CRC error checking, Full Duplex. But "autoconfigure" does not work. You must configure the driver manually. DF1Modem.pdf

[mjrx 1]

So it doesn't look like this will work unless the master is another compatible PLC running the DF1 radio modem protocol. I don't see how any 3rd party SCADA package would support this unless they specifically developed a "master" driver for it.

[Michael Lloyd 85]

I'm pretty sure Kepware has DF1 protocol. AES probably does. I use some little HMI's from Nematron (the free software kind) that have the DF1 protocol built in. It's pretty common

[Ken Roach 194]

"DF1 Radio Modem" is a variation on DF1 that's halfway between Full Duplex and Half Duplex. It was created for applications where the radio signal integrity is very good and less short-packet traffic for ACK and POLL are desired. While some third party software has adopted it, I've only used it with a PLC as the master. This application is standard DF1 Half Duplex. [soapbox] If you do radio or serial telemetry or ASCII applications with RS-232 more than once a year, the best money you'll ever spend is on an EZ-Tap from Stratus Engineering. These devices are so awesome I've actually sent them as Christmas presents.[/soapbox]

[mjrx 1]

Yes it is a very common protocol but even the app note you sent says "DF1 Radio Modem is a hybrid between DF1 Full-Duplex and DF-1 Half duplex protocols and is not compatible with either protocol". All the DF1 protocols I work with require and ACK and this protocol doesn't send one. I will test it and see. Thanks.

[Ken Roach 194]

This troubleshooting project intrigued me a little bit so I did some experimenting this morning. I don't have any MicroLogix handy, so I used an SLC-5/05 FRN 9. I'll try to do this without getting too deep into DF1 Half Duplex protocol, but here goes: Question 1: What conditions result in a "Lack of Memory/No ACK Sent" condition ? Question 2: What can we do about it programmatically ? Question 3: What causes it ? Consider an ordinary DF1 Half Duplex "Data Table Read". It consists of a five-part sequence: Command, ACK, ENQ, Reply, ACK. The following data is from a "Read from Slave #4, N21:0, 4 words" Cheat sheet: 10 05 = Enquiry (ENQ) 10 06 = Acknowledge (ACK) 10 15 = Negative Acknowledge (NAK) Master Command: 10 01 04 10 02 04 01 0F 00 C0 03 A2 14 15 89 00 00 10 03 33 C2 Slave ACK: 10 06 Master ENQ: 10 05 04 FC Slave Reply: 10 02 01 04 4F 00 C0 03 0B 00 0E 00 0E 00 0F 00 00 00 00 00 00 00 00 00 00 00 00 00 10 03 D9 F8 Master ACK: 10 06 This exchange results in the Messages Received, Polls Received, and Messages Sent counters in the slave MicroLogix all increasing by 1. Hypothesis: The Master ENQ is corrupted during transmission to the MicroLogix. This causes the Messages Received counter to increment by 1, but the Polls Received and Messages Sent counters don't increment. This data starts filling up a MicroLogix buffer. The Master should send another ENQ until it gets a reply. It has unlimited time to do so. If the Master just sends another Command to the MicroLogix, and follows with a proper ENQ, the MicroLogix replies to that command. But the old abandoned command is still in that buffer. After several* of these abandoned commands, the MicroLogix stops sending ACK packets in reply to new commands, and begins to increment the "Lack of Memory" counter. Because the MicroLogix is not responding to Messages with an ACK, this might look like being "locked up". If the Master sends an ENQ, the MicroLogix will respond. *several = I'm not sure. About 25 messages without an ENQ seems to increment the No Memory counter in my test. Is this a MicroLogix problem or a DF1 Master software problem ? Great question. National Instruments wouldn't be the first vendor (including A-B !) to build a DF1 driver that works correctly under perfect conditions but doesn't correctly handle abnormal ones. I think I'm getting too deep into this, so I'll cut to the chase: When the Lack of Memory/No Ack Sent counter C0:17 increments, execute a ACL instruction once to clear the Receive and Transmit buffers of the MicroLogix. The counter won't reset, but the MicroLogix will begin to respond to DF1 Master messages again. The logic I used for this was: "NEQ CS0:17 N21:9 BST ACL 0 YES YES MOV CS17:0 N21:9 BND". Copy and paste that into RSLogix 500. N21:9 is just a scratchpad register. Be sure you don't execute the ACL every scan; that will also cause the MicroLogix to stop responding to anything ! Edited 24 Feb 2012 by Ken Roach

[Michael Lloyd 85]

That looks pretty good (I really like their 485 widget) but when I download they free software and install it I get a missing DLL error. I realize that the software us useless without their tools, I just wanted to see the interface

[mjrx 1]

Wow. Very nice Mr. Roach. Your obvious expertise in this is greatly appreciated.

[mjrx 1]

Right now I am read/writing data from these files: B3, N7, T4, I:0 and O:0. I am thinking of consolidating everything into N7 in all of my PLC programs so I am working with one continous array in an effort to make my packet structure more efficient. I have in the past looked at many more data files than this, even with NI. I don't think its the NI DF1 driver issue because when I monitor my PLC progs with RSLinx running the DF1 Master Polling driver I often lose comms to the processor.

[Ken Roach 194]

Consolidating your data into a single integer file is a very good step; the number of serial packets will be reduced by a factor of five. At the same time you can implement the buffer-clearing logic if you like. The fact that different DF1 Master drivers appear to have similar symptoms, combined with the incrementing error counters on the MicroLogix controllers, tends to point to ordinary over-the-air data corruption. If you're not in a position to deploy serial analyzers, my recommendation is to include the CS0:xx values from each MicroLogix in the polled data, and store increments of the Error counters as events in your SCADA system. That will at least give you an idea of when the data loss is occurring, so you can see if it's correlated to events in the environment or the automation system.

[Ken Roach 194]

I was thinking about this today and remembering your comment about "DOX" transmission control. I realize that's sometimes a generic term but I know it's used by DataRadio modems. Can you specify the make and model of modems ?

[mjrx 1]

Its a CalAmp (formerly DataRadio) Gaurdian model UHF 450 mHz radio. I have sinced configured two of the four points for DOX and the other two as hardware handshaking to see if theres any difference, but there is no apparent performance advantage that I can see.