Sign in to follow this  
Followers 0
Chris Elston

GE 90-30 Rack, Loss of I/O Module Fault?

32 posts in this topic

I have a new GE 90-30 rack I've been working on. It's been powered up for a month now with no issues. All of a sudden today, I am getting all kinds of I/O faults. Actually, it's one fault, then it reset my profibus card which causes all the other faults to appear. I've been looking around for a manual that would help me decode the I/O table fault. I have attached the errors I have recieved....it did happen this morning, I just can't remember where to set the Time and Date in the PLC to the correct time...I'll find it again... LOL. Anyway, the fault I believe that starts this chain reaction is the first one: I/O Fault Table for location: 0.10 Loss of I/O Module It's at location 0.10. Where is that? Slot Ten? If it's slot TEN, then that is my profibus master. It keep resetting or dropping out. Any idea how I can troubleshoot what's it's dropping out for? I see both LEDs go out, then the bottom LED starts flashing, then it goes green again. It's like it's rebooting. It's been working fine for a week now, and I didn't change any configuration from yesterday (working fine) to this morning (not working anymore), keep dropping out about every five minutes. The machine is running idle with no I/O traffic, it just does it when powered up and idle. gefaults.pdf

Share this post


Link to post
Share on other sites
The 0.10 points to Rack 0 (the CPU rack), slot 10. A few years ago I ran into a similar problem with a Device Net master module. Periodically I would see a "Loss of I/O Module" fault table entry followed about two seconds later by an "Addition of I/O Module" fault. Occasionally I would see a "Loss of I/O Module" that was not followed by "Addition of I/O Module" and the Device Net network would go down. My case was ultimately resolved by firmware upgrades to the Device Net modules. For a short time, I had to replace some of the baseplates with older versions. It turned out that some modifications to the baseplates that were supposed to improve noise immunity created problems for the Device Net master module. I have no idea if you are seeing anything similar with the Profibus module, but I suggest getting your regional GE Fanuc Applications Engineer involved. If you don't know who that is, your distributor should be able to put you in touch with him (or her). The AEs have better access to the engineers in Charlottesville than the distributor's sales people. As a quick and dirty fix, you could try moving the Profibus module to the slot immediately to the right of the CPU module. You should also check the current draw of the modules you have in the rack. If you aren't already, consider using a high capacity power supply, especially if you are close to the limit of the 5 volt supply (15 watts) on the standard capacity unit.

Share this post


Link to post
Share on other sites
I talked to my GE guys and they recommend the same thing. I am seeing EXACTLY what you said above. Loss of I/O then Add of I/O. It's so intermittent. I am moving the cards around this morning, and will let you guys know. I am using the biggest power supply, the 330 unit. 30 Watts of 5 Volts. I also had the CPU crash this morning three times in a row, I had to flush it and reboot it to gain access again, then redownload all my software again. It crashed three times, then finally the third time it stuck. Attached is the fault I got. CPU Software Fault. Fatal..... M1270_fault_001.pdf

Share this post


Link to post
Share on other sites
I highly recommend taking Steve's advice and move the ProfiBus module to slot 2. Any 'Network' or 'Communication' module 'should' (not must) be as close to the CPU as possable. I say this because of the way the CPU communicates with the modules in the rack. A couple questions: 1. Do you see these 'errors' only while VersaPro is connected? 2. Are there any errors in the PB modules status area, and have you annalized them? If this were my system I would: 1. Upgrade to Proficy ME Version 5.5 and ServicePack 2 software. 2. Make dure that my CPU and PB module have the latest firmware. Check here: http://globalcare.gefanuc.com/ 3. Triple-check all PB connections and terminations. I can give detailed reasons if you need them, but mostly just "Best Practices".

Share this post


Link to post
Share on other sites
I have just completed the PBM to slot 2, and moved my CMM321 to slot 3, then shuffled all digial I/O cards down next. It's been running for 15 minutes and no PBM reboots yet, but the CPU crashed right after I changed it all over. I flushed it, redownloaded, and put it in run mode, and it's working ok right now, but it's only been 15 minutes since your post. 1. I am not using VersaPro, I am using PME version 5.50 SP1 build 3655. I don't seen any erros other than the CPU faults and I/O faults. When the CPU dies, or faults. I see a loss of communication because when the CPU dies I can't talk to it anymore and have to reboot. The ethernet port drops. 2. I have not looked at the status area of the PBM module. I looked once but I didn't dwell, because I didn't find anything helpful in the GFK2121A.pdf manual. I looked starting at page 4-1. It tells how to deceifer the I/O fault table but not much information about the status bits and what they mean. I know I have 128 bits of status I could be looking at, but I haven't found information or documentation that explains to me what they all mean. Do you have a link? 1. (again) Yeah, I see that am one service pack behind, I will do this. 2. (again) I can't tell what firmware is loaded in the PBM module. How do you tell this? I can't find any stickers on it or anything. I have a -CC after the part number, when means a firmware version, but I don't know which one. My CPU is CPU: 12.04 build 15A1, Daughter: 3.32 build 14A1. I need to figure out how to look that up online and see if that is the lastest revision or not. 3. Profibus was running two weeks ago with no ladder logic downloaded. Connections look good, termination resisters are installed in the proper locations. I double checked, but not triple checked. Profibus checked out with I/O toggling in the data tables just fine.

Share this post


Link to post
Share on other sites
My CPU faulted again, see attached error. It's the same fault. What I am noticing though is my IC693PWR330 power supply is trying to tell me something. The "OK" LED blinks two times, then pauses, then blinks six times, and starts over. "26"??? Does that mean anything? I looked up a manual GFK0898F.pdf on page 4-10, but it doesn't have details on what the blink codes mean. Any idea? Looked on global support for about 15 minutes, this is all I can find: CPU Blink Code Diagnostics ona 90-30 Power Supply How to read: Blink code 13 gives 1 blink on middle LED and 3 blinks on bottom LED. *( Two and three together is the reset or end of code ) 11 Flags or ALU failed 12 Bad Registers 13 Bad Stack mechanism 14 Bad Stack memory area For other blink codes contact gefanuc technical support. <<<----------------------------- M1270_fault_004.pdf

Share this post


Link to post
Share on other sites
Well, Ge Fanuc says I have a recalled CPU. The date code on my CPU is "621" and it needs to be send back, so for now, I have to wait for a new one.....sigh....I knew this was going to cause problems for me when I bought it. I had to sign a disclosure when I placed the order. I didn't think I would have to send the whole thing back. I'll keep you guys posted when I get it back, the new one, and if it starts working ok after that, I hope so.

Share this post


Link to post
Share on other sites
Sorry to hear this, I was hoping that this was going to be fixed with a firmware update. Or was your's one with a defective RJ-45 socket?Thanks for keeping us informed it may save someone else time.

Share this post


Link to post
Share on other sites
Well got my new CPU today and installed it, same problem. Now the problem is basically I am getting a PLC Software Fault. When I get this fault. The ethernet drops off and locked up the PLC. I have to power down, flush the memory, redownload to get it back. It appears to be the same error in the HEX codes but no one can tell me what it means.... M1270_fault_005.pdf M1270_fault_006_a.pdf M1270_fault_006_b.pdf

Share this post


Link to post
Share on other sites
Send all of this detail along with your case number to support@gefanuc.com and ask for Tier 3 support. If you need help with this please let me know.

Share this post


Link to post
Share on other sites
My "old" GE Supplier came in today and tried to help, I figured I would get them involved since I purchased the CPU, rack, and most of this system through them. It seems the problem stems from the power supply overloaded issue, I don't know if I have mention this in the past or not, I seemed to remmber asking and posting about it sometime ago. I have to admit however that I was under the impression that the power supply calculation in Machine Edition did not match those done by hand. I think now I was under the wrong impression...... I ran through this exercise when I first got PME, and haven't looked back since then. I knew that the power supply was overloaded and I knew that when I e-mailed my program to support at GE they were going to tell me this is the first thing I needed to solve. I am happy to report that I will now instead of "ignore" the power supply is overloaded issue by going with my gut feeling and start paying more attention to what is going on. I am NOT SAYING at this point the problem is solved, however I can tell you if the CPU would have faulted, it should have done it by now. The game plan presented by my ge supplier is to replace three input modules "655" with an equal or lower wattage input card "660" provided they are equal in performance. For now, my ge supplier suggested that I installed some other spare cards I have to "simulate" the load of the new input cards. Currently, I am now within 1 watt of the real load if I had the lower wattage input cards. So I am testing the system today. So far so good. The total wattage of the system is 27.6 watts. I feel like I newbie now, and I lost some of my pride posting this, but I felt it should be explained for any of those people that might run into the same issue. Hence my whole purpose of MrPLC. The 374 CPU is not to be messed with, it's power requirements are very important. Apparently those machines that I was pushing the red zone with in the past, had 350 and 364 CPUs in them, I assume they are not as susceptible to power issues in the rack.

Share this post


Link to post
Share on other sites
Sometimes you feel stupid and don't know why....I felt stupid enough to figure out why I got myself into this situation.... Here is the problem.... Remember I said that first time I changed over to PME, that I saw I had a red icon that shows the power supply was overloaded? And that I called my GE Supplier, and we both went through the calculation of all the I/O cards on the rack and came up with a different number than PME was saying the total load was on the rack? And that we had determined that when we manually calculate the amperage load of the power supply it appeared ok? Well almost on every single job I've done with a GE PLC, I also use a IC693MDL655 INPUT card. I like 32point I/O in the rack. Upon looking at the catalog pointed out by my ge supplier, we notice there is no rating for this card on the 24VDC side of the rack power supply. If you looking in the specs, you will only find the load on the 5VDC side of this card. Shown here online in ecatalog at 195mA. If you add this card in your project and check the power consumption, you will see your 195mA....but you also see a WHOPPING 5.38 Watts!!! So if you add 3 of these cards, you've consumed 15 watts of the BIGGEST GE power supply the 330 power supply. Because it's not printed in the catalog, nor is it printed online in the ecatalog about the 5.38 Watts of power it consumes, if you doing your I/O calculation manually, like we have done, your going to be wrong, and your going to assume your power supply is ok, and then someday your going to have problems with your CPU like I did. So there you go, the reason for my stupidity....Don't make the same mistake as me, and take a mental note. All other cards I used in my rack are documented correctly in the catalog and on ecatalog. Both power requirements for the 5VDC and 24VDC load on the rack power supply. IC693MDL655_datasheet.pdf

Share this post


Link to post
Share on other sites
It seems to me that the only time the module would use 24 VDC power would be if you used the PLC's power supply for your 24VDC for the inputs. If you use an external 24 VDC supply, you're not drawing from the PWR330. The 5.38 watts is equal to 32 inputs * .007 A * 24 volts. That would be a worst-case situation, all 32 inputs on at the same time and getting their power from the PWR330.

Share this post


Link to post
Share on other sites
I NEVER use the rack power supply to supply the 24VDC to any inputs. Never, never, never. I hope no one else does this that is reading this....break the habit.... The way this card is wired is using an external power supply. So with that regard, I don't know what the difference is, but these cards are out of my rack, and I installed some output cards and a junk card in their place with lower wattage and my CPU ran all night with no problems. Profibus did not crash either. It stayed up an running last night as well. I don't know why, I just know what works and what doesn't at this point. Cards out, WORKS, cards in, CPU Software Faults, which is related to over current I assume, so that is what GE is saying...

Share this post


Link to post
Share on other sites
I hear you. I've always interpreted the "Isolated 24VDC" as the terminals on the power supply module. If that's incorrect in the case of the 32-point modules, I'd sure like to know. Thanks for keeping us posted on your progress. I wonder if there is any way to measure the actual current draw, like a test point on the PS module???? Edit: Just to confirm, in addition to not connecting anything to the 24 VDC terminals on the power supply module, you are also not making any connection to pins B6 on either of the connectors on the MDL655. That would be to terminal 11 if you are bringing the cable out to the TBQC or the gray/black wire in a CBL327 or CBL328. My understanding is that the B6 pin is an extension of the 24 VDC isolated supply. Edited by Steve Bailey

Share this post


Link to post
Share on other sites
Correct. We don't have anything connected to the following: B5 = terminal 9 B6 = terminal 11 B7 = terminal 13 A6 = terminal 12 A7 = terminal 14 A8 = terminal 16 We have our commons (for the inputs, which is NPN) ack! connected to: A5 = terminal 10 B8 = terminal 15 All rest are input connections.

Share this post


Link to post
Share on other sites
Just an update...and the sagga continues. My system ran without any CPU faults for 1 1/2 days, now I am back to the original profibus master reboot problem above...sigh...this is really whacking me out. I did allot of troubleshooting yesterday, and what I did was remove 5 slave nodes. I have been running well since I removed the nodes. I have a total of 24 slave nodes, minus 5 right now and no profibus reboots. I did enough troubleshooting yesterday to determine that it's not a RACK power issue. I am 100% convinced that is ok based on swapping and removing of I/O cards I did yesterday. So now that I have established a good baseline today with the machine running just fine today, I am going to start adding slaves back into the system one at a time. I think it might be a throughput issue but I don't know for sure. I am using my devicenet experience, by using the SLOWEST baud rate of 1.5 mbs. In devicenet the faster I went, the more problems I had with my systems (those where Allen Bradley PLCs). Maybe I shouldn't apply that same logic to a profibus system???? Anyone have any experience with bauds speeds in your profibus system? Your next question to me is how much data am I exchanging through my slaves, most of my slaves exchange 4 words of inputs and 4 words of outputs. One slave, a servo module exchanges 16 words INPUT and 16 words OUTPUT. Everything else is either 1 word or 2 words. If you add all that up I am not even close to the limitation of the master card, which is somewhere, around 3000 words. Next your going to ask me what my power is to the profibus. I have a DEDICATED 24 VDC power supply that supplies power to the profibus, it's separate from the rest of the machines outputs. Obviously, that power supply supplies power to the inputs, like photoeyes and laser sensors, but no outputs are on this power supply. The power supply is a 5 AMP supply, and draws 1.97 AMPS. The voltage of the profibus power is 24.22 volts, when the module reboots automatically, the voltage increases to 24.28 volts, then drops down to 24.22 volts when the reboot is complete, then the slaves are re-enabled. Does anyone else have any other ideas what might be the issue with my profibus master card? I am experimenting today with the TOTAL number of slave nodes, but I guess I would like to hear some input regarding network speed. Do you think I could have a BAD SLAVE that is causing my reboot issues? Why would a bad slave cause the entire card to reboot?

Share this post


Link to post
Share on other sites
Update. Two days ago I was running my system with 19 slaves active, I ran most of the day like this and left the machine running all night with 19 slaves active with no errors or reboots. Yesterday, I disabled 3 of the slaves I had been running, and enabled 3 slaves that had not been active before, so I changed a few slaves around, but left the total active slaves at 19. I ran half the day like this and no reboots. Generally I would have seen a reboot by then. After lunch yesterday, I went ahead and added back my missing slaves, except two of them. The servo drive, and another XY controller that the customer has, so at noon yesterday I had 22 slaves active again. I've had this active before and it worked fine for awhile, then crashed after a day and a half. Of course, it ran from noon yesterday until 5PM without any problems, no reboots. I went ahead and left it running all night. This morning I come in, and profibus is crashed. 2:30AM, see log attached. (starts at page two in the PDF) I got something different on the profibus card. It's a blink code I don't understand. It goes like this. COM LED - 2 green color blinks COM LED - 6 red color blinks SYS LED - 1 long red color blink blink code recycles...... I unplugged the communication cable right at the profibus master card, there was no change in state of the blink code. In other words, unplugging all the slaves via the profibus cable didn't effect the error state the profibus card had gotten itself into. I powered down the PLC rack, and powered back up the PLC, that card came back to life again. No errors so far. Currently I have 22 slaves active. I'll try and run like this today, with 22 slaves active, this is normally when the card starts acting up. I think it's time to replace the profibus master................. M1270_fault_007.pdf

Share this post


Link to post
Share on other sites
Update #2 Like my last post, I have 22 slaves online. I still have the servo drive down (offline) and an xy servo offline. Profibus still reboots, maybe once every 3 hours with 374+ CPU. I have not replaced my PBM yet, still waiting on snail mail to deliver the goods. I should have it today. (Monday) As a secondary test, I was able to get a borrowed 350 CPU from my GE supplier. I converted my program from the 374+ CPU to a 350 CPU, installed the CPU, cleared the memory, downloaded my application. The first thing that happened was a software CPU fault. I've gotten this on the 374+ too, right after the first time I download after a memory clear. But then normally on the 374+ I just do it again, and the program "sticks" the second time. This was also the case on the 350 CPU. I re-downloaded it one more time, cleared the faults, and left the system in a run mode, (PLC ladder logic scanning) ALL WEEKEND LONG. This is the exact same thing I do when I was testing my system before for an overnight testing during the week. I came in this morning on Monday, and I saw the PBM had not rebooted. I had ZERO faults in the CPU side, and ZERO faults in the I/O side WITH 22 slaves nodes online which I have not been able to do before. I find this very interesting, it appears from this test, that there is still issues with the PBM module and the 374+. Of course, I wonder if GE is going to turn out like Allen Bradley, I will now attempt to tell GE what I found out. Will they be receptive to my findings? I must not be qualified to report these findings. As always, I'll keep you guys posted. You guys have to chime in here every now and then so I know someone is reading my posts, so I don't feel like I am talking to myself... LOL... So now I have a question of experience to others, should I have to replace the 374+ with like a lower CPU say a 364 or 350, I will lose my ethernet. However I have a CMM321 card. I bought this card for Modbus TCP/IP reasons to talk to my Cognex Cameras (dedicated). Should I need to worry if I put my HMI screen on this same card? I shouldn't have any problems talking to my HMI screen and my Cognex cameras should I?

Share this post


Link to post
Share on other sites
Have you considered (or has your distributor suggested) the CPU366? That's the CPU with the built-in Profibus Master. That and a CMM321 will take the same amount of space in your rack as the CPU374 and the Profibus Master module. I think the CPU366 is the right number. My hard copy of the catalog has a slight discrepancy in the descriptions of the features of the CPU366 and CPU367. I have used CMM321s on a network with multiple drops using SRTP protocol without any problems, but it was one computer running Cimplicity talking to multiple CMM321s, not one CMM321 talking to multiple nodes using ModbusTCP. And yes, even though I haven't posted recently, I've still been following your posts to this thread with interest. This is kind of off-topic for this thread and more of a question for Russ. Are the Profibus modules for the 90-30 still made by Horner Electric? I ask because the problems I mentioned in my earlier post were with the Horner DeviceNet modules and over the course of this summer I've lost confidence in Horner products. Lots of problems with the Horner Genius interface on GE/Fuji AF300P11 drives.

Share this post


Link to post
Share on other sites
I didn't know such a CPU existed.... Wonder "why" they made such a product; wonder if they wanted to be SURE that it would talk to each other without any issues. FYI: I've done several profibus systems with GE 90-30 and Horner cards in the past. I've used 350 CPUs and 364 CPUs without a hitch and a Horner PBM. I actually bought a Horner card for this system, but then I found out that GE won't release information to Horner that they need to allow the 374+ CPU to work with their PBM master module. I think that's pretty crappy of GE. I don't know what their logic is behind this. It's either evil, (we don't want Horner to take away our business because we have our own PBM card, or it's good, we want to control the firmware process of the CPUs and "OUR" PBM card totally so that there will be no compatibility issues for the end user....) Guess what, it appears initially here, there is some compatibility issues. I also think Horner has one of the most upstanding tech support people on the face of the planet. I am always pleased when I talked with Horner in the past; their tech guys are A-1 top notch. Unfortunately this time, I couldn't use their product.

Share this post


Link to post
Share on other sites
HOT OFF THE PRESS BOYS AND GIRLS! Looky what I got (read PDF).....Exactly my issue. Whitepaper is three days OLD, released 10-20-2006. I am in the process of getting a firmware DD profibus card right now (think that means a new EEPROM chip or new card).... I wonder what brought this on? Was GE on top of this issue? Or did I push the right buttons? PBM200 May Stop Scanning PROFIBUS Network IC693PBM200-Cx and earlier PROFIBUS master hardware may stop scanning the PROFIBUS network for a period of time and then recover. When this issue occurs the SYS LED on the PBM200 turns off, and the COM LED blinks green. There is a Loss of I/O Module fault in the PLC CPU’s I/O Fault Table, and the PBM200 does not scan the PROFIBUS network. Depending on the PLC CPU’s scan time, this condition persists for several seconds to several minutes. Then, the SYS and COM LEDs on the PBM200 return to a normal operational state, an Addition of I/O Module Fault is recorded in the PLC CPU’s I/O Fault Table, and the PBM200 resumes scanning the PROFIBUS network. This issue is corrected in IC693PBM200-Dx and later hardware. It cannot be resolved with a firmware upgrade. Customers with IC693PBM200-Cx and earlier hardware who experience this issue should contact GE Fanuc Technical Support. GFK2119D.pdf

Share this post


Link to post
Share on other sites
Sounds like good news for you. It probably means that you're not the only person to experience the problem. You were just fortunate enough to still have the hardware in your shop and not at your customer's site. As much of a PITA as it has been to get it resolved, think how much worse it would be if you had already shipped the system to the end user. Let's hope it really solves the underlying problem and isn't just a "duct tape" fix.

Share this post


Link to post
Share on other sites
I agree, there is always a more positive note. We have a really good customer for this machine though, they have a very good technical staff, (better than most companies we work with....) and I am sure they would endure the pains with me and understand the issue until it was resolved.

Share this post


Link to post
Share on other sites
44A752220_G03.ZIP For your convience.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0