craisondigital

Program freezes suddenly after about 30 seconds

23 posts in this topic

We have an auto-clave unit purchased at auction roughly 4 years ago.   CompactLogix L32e and PanelView Plus 1000.  All has been working for us until last week.  Suddenly, the program seems to freeze up after around 30 seconds.  I can still touch the panelview and the touches are registered, (the mouse icon moves) but the program does not respond and I can tell the program is frozen because the clock which displays seconds stops counting.

When i boot into safe mode (clicking white box on boot) the system does not seem to freeze at all.  I am able to navigate those menus w/o issue.  We don't have any of the software to connect to the PLC.  I understand this is a somewhat broad issue, but does anyone have any idease where i can start in troubleshooting this? or is their any further info i can provide which would be helpful?

Share this post


Link to post
Share on other sites

Is the PLC program running or is there a fault light?

Do you have numeric values on the PV that change state, like showing question marks or error?

You mention that the clock freezes, is the clock using the PLC clock or the internal clock of the PV?

Seems like you might be losing communication with the PLC.  You should make a backup of the MER and port settings (Ethernet and Serial Port).  You will then probably have to reload the firmware.  THen copy your MER and port settings back into the PV and test.

Share this post


Link to post
Share on other sites

The white box on the PanelView Plus 1000, during startup, takes you into Configuration not Safe Mode (like a PC).  A PanelView is not a PC...it is a dumb terminal that communicates with the PLC to get updated datapoint values.  PView configuration is for editing communication parameters, calling up a startup program, internal settings edits.

How is the PView Plus 1000 communicating with the CompactLogix L32E (Ethernet/IP or RS-232)?

PCMcCartney1 is correct...move onto looking at the status lights of the PLC.  If the PanelView shows numeric data that becomes # symbols when it freezes, the PanelView is not getting the data.  Refer to pg. 131.  http://literature.rockwellautomation.com/idc/groups/literature/documents/um/1769-um011_-en-p.pdf

After the system freezes up, how do you get it back in operation again?

Share this post


Link to post
Share on other sites

I've seen situations where the PLC program and it's numeric values are not bounded or limited to their data types.  In otherwords, the numeric input is allowed to enter a value greater than 32767 oe less than -32767 for an interger values.  The moment the PLC sees this, it faults.  Usually, I have to go online with the PLC and review the tag database and find the offending value, change it to an acceptable value, reset the fault and place back in run mode.  Occasionally, you will find logic, like a compute block whose result exceeds the range of the INT or DINT.  Other times, it's a violation of the array subscription.  Like a dynamic pointer to a specific value in an array that is outside the length or size of the array.  All of these could be positive or negative values.  Properly written code, in both the PLC and the PV+1000, should prevent this.

My point about reloading firmware is more along the lines of a running PLC and therefore the problem is localized to the PV+.  I have had rare cases where a power spike has corrupted the firmware or more specifically the ethernet driver of the PV+.  This requires "updating" the firmware.

Share this post


Link to post
Share on other sites

Thank you for the info.  The values do not turn to question marks, but they freeze as well.  For example, I've tried heating a probe after the freeze and the value does not change.  When i open the door, during the freeze, the door open icon does not appear.  I'm not sure which clock the program is using. I am connected the plc to the panel view via ethernet cables.  To get back to operation again we have to power down the machine and power it back on.  Once the program loads it freezes about 30 seconds later.

I'm going to check the status lights on the PLC now and will report back. 

Share this post


Link to post
Share on other sites

Presumably, the PLC and PV+ are communicating via EtherNet/IP.  There might not be anything wrong with either of them.  Is there a hub or switch in the panel?  Have you changed ports?  Have you replaced cables with "known good cables"?  Have you attached a PC in the same subnet and tried pinging the IPs for the PLC and PV+?

Share this post


Link to post
Share on other sites

ok, yes their is an industrail switch.  I never thought of hooking up a PC and pinging.  I assumed I would need the special software.  I'm going to try that in the morning.  I will start with that, and if not, will try changing ports and cables.

Something else I just thought of.  I have a backup of the .mer file from when we first started and everything was working.  My thought was, that maybe if I reupload that file, it might solve the issue (if it was in fact an unacceptable integer being placed somewhere).  I believe I can reload the .mer file by going into config mode with a usb flash stick.  Any thoughts?  Think its worth a try?

Share this post


Link to post
Share on other sites

It might.  So now you are saying it used to work, but you have made changes to the MER, what did you change?  What else has changed since the original MER was on the PV+?  Have any settings been changed like the subnet masking or default gateway?

Edited by pcmccartney1

Share this post


Link to post
Share on other sites

When I originally received the unit it did not work, but I detected the issue was with the PV+ and replaced the PV+ with a unit from ebay.  Then I loaded the .mer, changed the ip to match the subnet of plc, and everything worked fine for the last 4 or 5 months.  The freeze just suddenly occured, but now happens each and every time..

To my knowledge nothing has changed with the .mer and also no settings have changed either.  I'm not sure in terms of the .mer if the technicians are changing values like acceptable temp ranges, etc, but do not believe so.

Share this post


Link to post
Share on other sites

Ok, I can confirm, I do not receive any error lights on the PLC after the freeze.  Also, I do not believe the problem is with the switch or the cables.  I disconected the ethernet cable from the PV+ and booted the machine.  the program loaded, with the ? as their was no comm w/ PLC, but the freeze still occured.

Tomorrow, the plan is to first try to reload the .mer.  If that doesn't work, i will hook up a laptop to the switch and try pinging the PV+ after the freeze.  Please let me know if you can think of anything else.  

Also, i know the PV+ has an internal flash card and also some memory.  Are these known to crash at all? 

Share this post


Link to post
Share on other sites

Yes, the internal flash card is where the firmware is installed and also handles the comms drivers.

Share this post


Link to post
Share on other sites

No, rare failure.  Might have happened to me times in the last 15 years.

Share this post


Link to post
Share on other sites

i can confirm the following..

When i try to reflash the firmware, the message pops up asking me if i want to install the firmware, I choose yes, and the screen turns white and says upgrading firmware please wait.  I left it over a half hour, but the same screen was up so I assume it froze during the upgrade.  When I repower the machine, my program loads but the same issue as before..

When I try to reinstall my mer file, and I hit the copy button (from config mode), it freezes up.  When I repower the machine, the new .mer is their however will not load. (It freezes when i choose to load it).

Only thing i can think is that it is the internal flash card acting up.  Are replacements available or do i need to change out the entire panelview.  anyone have any other ideas?

Share this post


Link to post
Share on other sites

Talk to Kaiser_Will or MrPLC (this forum) has a store.

Share this post


Link to post
Share on other sites

I assume you have zero Rockwell/Allen-Bradley software (no Studio5000 [PLC], no FactoryTalk View ME [HMI]).  If you do not have FTView ME software, you will be unable to change to a different HMI model and convert the application over; you could contract someone with the software to convert the application.  

Depending on the number of HMI screens, pop-up screens, embedded scripting, and if you do not possess all of the necessary Rockwell software, it would be in your organization's interest to replace the HMI unit with a manufacturer that has a price point that fits in with their objectives.  To have an important process machine for 4 years with no supporting software is high-risk.

I am not 100% that you have identified the root-cause of failure; you may be treating symptoms and now the system is all boogered up.  You noted that after purchasing the used process machine that it was non-functional, the PanelView was suspected to be the issue, was replaced, ran for 4-5 months, and now the same (?) issue started occurring (data is not refreshed on the PanelView, machine becomes non-operational).

What firmware version are you trying to load in this PanelView Plus 1000?  What firmware version was in prior to your attempted firmware update?  What is the PView Plus 1000 model part number?  Did you use the Firmware Update Wizard - scratch that, it only comes with FTView ME.  Where did you get the firmware update from?  Can you verify that the properties of the firmware update file is unprotected)?

Share this post


Link to post
Share on other sites

im not sure it's quite the same problem, but similar..

When the machine first arrived, the PanelView would power on, run through the DOS screens, attempt to load Windows, but Freeze before loading windows (way before loading the application) at the very first blue screen, and so all we were left with was the blue screen and an hour glass.  We found a Tech about an hour away with all the Rockwell Allen-bradley software and had him come out to look at the machine.  He was on the phone w/ Rockwell support for some time but they were unable to determine the problem.  He attempted to upgrade the firmware, but it was not recognized, however he left me a copy of the firmware on a USB flash drive.  This is the copy I have now.  I can upload it to this forum if its ok?  I'm not sure exactly which version it is.

The tech (and rockwell support) recomended replacing the PanelView, but the quote was around $4000 which the company did not want to pay.  Instead, I ordered a PanelView Plus used from Ebay for around $400.  When the unit arrived, I attempted to load my .mer file, but it told me that I first needed to update the firmware.  I updated the firmware, by placing the files on a CF card, and inserting that into the PanelView while it was loaded into config mode.  A prompt came up asking me if i wanted to upgrade the firmware and i choose yes.  The upgrade was successfull and I was able to load my .mer file now also using the CF.

At this point, the application loaded, however we received ? instead of values.  We had the Tech come back out, who was able to use his software to reload the program on the PLC.  Ever sense then (5/2017) everything worked.

Now suddenly we are experiencing a similar issue, EXCEPT, now Windows loads with no problem and also the application loads with no problem however the freeze is occuring after the program is loaded, typically when the user is logging in or starting to run a program.  the unit also freezes while in config mode now, when trying to copy over a new .mer file, or upgrading the firmware

Share this post


Link to post
Share on other sites

I found this Knowledgebase article (access level: Everyone):

https://rockwellautomation.custhelp.com/app/answers/detail/a_id/866399/page/2

You will need a login (free) to view the article.

They have another couple of articles that require a TechConnect plan. The first covers when the screen freezes so the buttons don't work but the data is updating. The other one says that the cause was a network cable loop (a cable connecting 2 switch ports together).

 

Share this post


Link to post
Share on other sites

Thanks Joe,  that article does seem to describe my issue exactly, however I realized I am on the PanelView Plus 1000 2711P which has firmware 5.10.16 which seems to be the latest available for this model.  To the best of my knowledge, the patch is for a CE version which needs a firmware of 7.0.0 or above to apply this patch.   I've attached a picture of the back of my panelview.

20180130_155445.jpg

Share this post


Link to post
Share on other sites

You have a PanelView Plus 1000 HMI, requiring FTView ME software.  Check.  

When the unit freezes, and it sounds repeatable at this point...it will happen.  When the unit freezes, take note of the status indicators on the back and post the condition.  Refer to pg. 166.  http://literature.rockwellautomation.com/idc/groups/literature/documents/um/2711p-um006_-en-p.pdf

Have you serviced the internal HMI battery?  This watch-style battery, I believe it is a common 2032 (available most anywhere).  Refer to pg. 133.  http://literature.rockwellautomation.com/idc/groups/literature/documents/um/2711p-um006_-en-p.pdf

Share this post


Link to post
Share on other sites

I think all of the PV+ units are really Windows CE systems that they've locked down very tightly so you can only interact with their own front end. The Siemens MP (and new Comfort) HMIs are also CE boxes but they present the actual Windows OS environment to you. Considering an issue or two we've had with folks messing around in the Windows side, I think I prefer the Rockwell way.

Share this post


Link to post
Share on other sites

Thank you for the information.  I checked the voltage on the HMI battery and it was 2.99 so still seems ok.

The status indacators are another story.  When I turn the machine on, the two lights come on and stay lit only about 3 seconds during startup.  The fault light turns off, and the comm light stays on for only another 2 or 3 seconds and then turns off as well.  So both lights are off and stay off well before the application loads.

According to the manual, it says that if both lights stay off, to check the power supply.  The power supply seems to be putting off steady 23.8 volts dc, and so the manual says to replace the logic module.

Share this post


Link to post
Share on other sites

UPDATE:  The logic module arrived from ebay, I installed it, and all appears to be working again!!

1 person likes this

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now