NX7 PLC Fatal Error

[kckku 9]

We have had two fatal PLC errors in the last two days. Both times nobody was interacting with the PLC and the machine was idle.

First instance:

• Machine was left in an E-Stop state. Work was being done over the weekend and the guys didn't want anyone attempting to run the machine.

• Arrived the next morning and was able to connect to the PLC

• Not 3 minutes after connecting, all safeties were dropped and we lost connection to the PLC

• PLC Error LED was solid red, signifying MAJOR FAULT

• Upon power cycle, we were able to connect to PLC to see Major PLC fault code (100A0000 hex). This means 'Data not saved to battery backup memory'.

• Upon viewing data on plc, we noted none of our RETAINED variables were present
  • ALL RECIPE INFORMATION WAS LOST (Servo positions, camera job ID’s, etc.)

• The only way to clear the fault was to download the existing offline program to the PLC and performing a power cycle

• All retained data had to be restored from backups and some had to be typed in manually. Some servo positions had to be retaught. Overall it took a day of work.

Second instance:

• Another fatal error. This time error code 40030000 hex. This means 'PLC system processing error'.
• Nobody was working online with the PLC. 

Not sure if anyone has had these happen before? So far we have swapped out the PLC with our spare unit. Also trying to get some feedback from Omron to find out what happened. They have stated that they need to reach out to Japan to get more details. Overall a terrible experience.

The PLC is a NX701-1620 with version 1.21. The Sysmac Studio version we are using is version 1.45.

[skyfox 5]

Not familiar with that PLC. Just a guess. According to the error message above, it states unable to to save to battery backed up memory during a POWER-OFF condition. Was the power off done by someone or did the system lose power? Could it be because of a bad battery? 

Edited 4 May 2022 by skyfox

[IO_Rack 181]

22 hours ago, kckku said:

All retained data had to be restored from backups and some had to be typed in manually. Some servo positions had to be retaught. Overall it took a day of work.

We had a similar issue with the retained data but no fatal error on an NJ501. Sorry I don't have an answer for you but want to share this similar experience. 

- Machine was powered off for 3 days prior to the incident. Happened on power up. Battery is good. Additionally, the battery is only for the Time/Date. 

- There was a large time/date gap in the error log. (Months). No abnormal PLC errors. We don't know if the time was changed but suspect it wasn't since there is no method to do it remotely.

- Techs found a shorted sensor. Not sure if that had anything to do with it since this is a common issue with our machine due to the physical motion.

- We were able to restore our memory from a file. I would highly suggest you perform a memory backup. Tools > Backup > Backup Variables and Memory.

 

We were going to contact Omron about the issue but we couldn't find enough information to develop questions. 

[kckku 9]

15 hours ago, skyfox said:

Not familiar with that PLC. Just a guess. According to the error message above, it states unable to to save to battery backed up memory during a POWER-OFF condition. Was the power off done by someone or did the system lose power? Could it be because of a bad battery? 

I would think the battery should be good. This is a new PLC that has only been in use for over a year or so. The vendor came back to the plant to do some additional retrofit work and clean up over the weekend while production was down. I really don't think anything abnormal happened during this time. They have been doing this on/off for some time (last couple of months). 

Also not quite sure what exactly is battery backed up memory other than the date, time, and retained memory variables. There isn't anything really on this machine that is different than others in our plant. Just a dial table with a bunch of 1S servo axes, 2 or three SCARA robots, and a couple of ABB 6-axis robots. It's really a mystery.

Omron hasn't responded as of yet but it's only been a few days. For now we swapped out the PLC with one that we have as a spare. So far nothing bad has happened yet. Of course management wants to know what exactly happened but we really don't have an answer. Lost production for like 1.5 days.