Sign in to follow this  
Followers 0
Ken Moore

CLX L61 Lossing it's mind?

21 posts in this topic

Got a call last, our CLX L61 SIS PLC had tripped for no reason. I had the night supervisor open the PLC panel and tell me what the status of all the cards were. All the I/O modules and the PLC were flashing red. I had him cycle the power, everything came backup green, except for a ProSoft Modbus module ( used to send status words to our DCS). I crawled out of bed, drove to the plant, and tried to go online with the processor, I knew something was up when I had to reconfigure my DF-1 driver. The PLC had lost it's programming, so I downloaded the program and everything started up and ran like expected. Now the big question, what happened? In the past I've had AB PLC's loss there programming due to power issues. But this PLC is connected to an industrial UPS (12.8 kva). The UPS in question supplies power to several PLC systems, and only this one had any issues. However, this is the only CLX PLC, the other systems are PLC-5's, SLC's and TI505's. We usually have power issues due to local weather, but everything was calm tonight. Nothing else in the plant was affected. The UPS has an internal log file, which I plan on downloading Monday, perhaps this will point me in the right direction. This is our first CLX system, we have resisted upgrading due to software cost, training, spares etc... So, to prevent this in the future, could I just install a compact flash card? Can the CLX platform be configured like the EEPROM on the SLC's? I think I read somewhere that the program could be auto loaded from the card, but not firmware. So if we had to replace the processor, plugging in the CF card wouldn't work, but if we just lost programming it would? BTW, since this is an SIS system, we are running version 15.04 of 5K, when this system was installed version 16 had not been certified, don't know if it is now or not. But, with our safety policies, if I upgraded the firmware, I would have to re-validate the system, which would be tedious at best. Has anyone experienced similar problems?

Share this post


Link to post
Share on other sites
I have had a L61 loose its program before, it would loose its program randomly over a 3 month period. Changed the power supply but this did not fix the problem. Changed the processor card and the new unit has not faulted since. When looking at the old card there was some corriosion on the pins of one of the chips, I do not know what this chip does on the board. Alan

Share this post


Link to post
Share on other sites
I've had a CLX system drop it's program for no apparent reason, AB tech support advised upgrade the processor firmware which was done and it has been OK so far. The processor was on v16 before the upgrade.

Share this post


Link to post
Share on other sites
Recently there was a service bulletin that went out from AB. It lists CLX processors, ENBTs, DNBs, and maybe more that have known chipset issues that can cause what you are experiencing. I don't have the info in front of me, but Rockwell is replacing these if yours match model # and production date.

Share this post


Link to post
Share on other sites
I would strongly advise calling Rockwell and getting the Fault Extraction Tool from them before flashing any firmware. It may or may not still hold some clue to what happened

Share this post


Link to post
Share on other sites
On a ControlLogix controller the CompactFlash card actually holds both the firmware and the application program. There's a descriptive file on the CF card that the controller always reads at powerup, which includes the settings you make when you "burn" the CF image. In this sort of case you would want the card read only in case of invalid memory (= corrupted memory or just blank RAM). This would allow you to cycle power on corrupted controller, and load the OS and program back into memory. In the event of total controller failure you could take any 1756-L61 controller off the shelf, drain the memory, put in the CF card, and power up. Eventually you'll probably migrate to v16, which includes the Firmware Supervisor feature that will allow compliant firmware to be automatically loaded into replacement modules (network, I/O, motion, etc) as well as the CPU. Call RA Tech Support and go through the memory dump utility and rule out any firmware, application, or hardware problems with the system. Edited by Ken Roach

Share this post


Link to post
Share on other sites
Well...... Happened again tonight. I told them not to touch anything before I got in. A member had PM'd me the diagnostic dump, so I was able to get a dump from the controller, will be contacting RA support in the AM. I went online and had a Type 60 major fault. Here's what the help files have on type 60: An unrecoverable major fault has occurred. The controller has reset and cleared all of its memory. A RAM image downloaded to CompactFlash was not successful. Contact Technical Support for assistance. At least this rules out the UPS system. This processor has only been in service about 14 weeks, kind of disappointing.

Share this post


Link to post
Share on other sites
I've found the email info from my CLX losing program problem: "I've spoken to rockwell tecnical support who have told me the following: The major fault is something processor related and not caused by software problem (e.g. indirect address looking at data table that is not created etc) They have recommended upgrading the firmware to version 16.20 (latest version) I will send you the file tomorrow. (you are on 16.07) Unfortunately this seems to be the Rockwell stock answer, have you got latest software / fimware? If you answer no, they recommend updating straight away. Looking at the manual, when a major fault code 60 occurs you should examine the status of the OK and RS232 LED's and make a note, this info will assist in diagnosing the problem with tech support." As I said earlier this has not occurred again after the upgrade which is now 5 months ago. Edited by ratcliffe_ic

Share this post


Link to post
Share on other sites
Well, this last time it only ran for about three hours before faulting again. I replaced the processor with a spare. Hope this takes care of my problem, now I just have to get a warranty replacement. ( I hope.) Upgrading the firmware maynot be an option, the last SIL rated firmware I could find referenced is 15.4.

Share this post


Link to post
Share on other sites
May or may not help, but I've had that exact problem twice... ( on one line to the point, I kept 3 processors loaded with the program, so night maintenance could just swap them in one at a time, and let me get my sleep.) The problem turned out to be a bad RACK... the backplane itself. I don't know why, but replacing the backplane made stopped the CPU's from faulting and losing their memory. You might want to give it a shot. Aside from that, we've had no troubles with any of our CLX based systems.

Share this post


Link to post
Share on other sites
Friday at 5 pm, our sole Logix L61 CPU lost its program with a non-recoverable fault. It had been running for six months prior with no problems. I looked at the fault tab when going online, but did not investigate further, just dumped its brains back in and put the machine back into production. I ordered a 1784-CF64 compact flash card for it today... It is running firmware 16.07, no fancy code, a very simple machine with ENBT to flex I/O and PV+ and some discrete and analog I/O. My question to Ken Moore: Did that happen after you had installed a CompactFlash card? I am wondering if the memory card is going to do me any good, and maybe I should update firmware as well. Thanks, Paul

Share this post


Link to post
Share on other sites
Contacted RA support, sent them the two dumps and the .ACD file. Was told it could be up to a week before I hear anything. I'll let the board know what I find out. Ken

Share this post


Link to post
Share on other sites
Paul, Haven't installed a compact flash yet, was just wondering if that would be a viable backup. So that operations could just cycle the program/run key and I get to stay in bed until the AM. Ken

Share this post


Link to post
Share on other sites
Yes, that is what I plan on doing. I think I will also add a counter in the logic to detect the number of occurrences of loading from the memory card too, so that I can tell if it actually ever gets used. Thanks for the reply... Paul

Share this post


Link to post
Share on other sites
I have actually experienced a memory clearing error before. Luckily it was found during testing of the code. What was found was that the code was attempting to access an array area which was not defined (i.e. going over the size of the defined array). Placed some error trapping code in that would prevent this. All worked out fine once the code was debuged and finalized. Never had this problem again.

Share this post


Link to post
Share on other sites
Just got an email for RA support, not very helpful. They do not appear to know anything about the fault. I will call them Monday and discuss.

Share this post


Link to post
Share on other sites
Called RA support one more time, got a better person on the line this time. The controller has a bad Atmel chip, and will have to be replaced. If you look down inside the module ( from the top) , you can see the Atmel chip, it's about 1 inch square, if the second line of numbers ends in 64, you may eventually have the same issue, if your chip ends in 65, you're okay. The processors will run for an extended period without issues, then start to develop the symptoms I posted, the time between events gets shorter and shorter until finally the processor will not work at all. thanks everyone, Ken

Share this post


Link to post
Share on other sites
I had the same symtpoms. Each time, the length of time between failures became much shorter. We went through the same fault dump routine. I can say the final Tech Support engineer was very helpful. Rockwell tech support sent us the attached Product Service Advisory detailing a recall of ControLogix products manufactured between October 2006 and May 2007. The PSA describes how to check the Atmel chip for a serial number ending in 64 (bad), or whether there is a white or silver dot in the upper left corner (good). What I've found, though, is that Rockwell does not do what they publish in the PSA, and my correspondence with them over the PSA mentions that due to the large number of products affected: DO NOT expect to receive a NEW product although mentioned in the text that reads "...for information on how to exchange your Logix product for a new one." DO NOT expect to receive a replacement product that will "...have been manufactured in the last six months." Another thing I wasn't happy about was that the L61 was shipped from the remanufacturung facility via UPS, with no other packaging except for the original factory cardboard box that contains the processor. I don't feel that method is prudent for shipping a multi-thousand $$ electronic controller. Rockwell hasn't answered me on that one yet. ControLogix_PSA_2007.pdf

Share this post


Link to post
Share on other sites
With RA, "new one" means new to you. They usually ship a used, but repaired unit. My understanding is that they are replacing your used broken unit with a used functional unit. I've never had an issue with this policy in the past. But, in my case, the unit that failed is one I got from them about 4 months ago. You would think that they would not ship "repaired" units without applying all known "fixes".

Share this post


Link to post
Share on other sites
This past weekend, one of my L61s displayed an unrecoverable fault code 60 and lost a program. After a little digging, I found several others that had failed over the past year (including some ENBT modules) with the same fault. I was told that we returned 30 some modules several years ago; but, a few more must have been floating around. The information here saved me a lot of troubleshooting. The PSA attached above led me to a related issue described in Tech Note 41204. This note extended the date range from May 2007 to June 2007. Thanks 1192625040.pdf Edited by jbond

Share this post


Link to post
Share on other sites
That's the whole purpose of the board, users helping users. Glad we (Royal WE) could help. Ken

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0