Ethernet & grounding

[ssommers 20]

Hi all - I was asked an interesting question by the IT manager... What kind of grounding is needed for an Ethernet system that runs between 2 buildings? Here's the background: This is a small plant with 2 buildings - office & production. The servers are located in the office building. The central ethernet hub is in a very messy wiring closet in the office building. The office & plant networks are connected through an underground conduit with cable that may not (?) be rated for underground use. Both buildings are served from the same incoming power & have a common electrical ground point. What's driving this? The IT manager wants to have an outside firm clean up the wiring closet while enlarging the computer room and adding some ethernet receptacles. The estimator from the outside firm told him that there's a code issue (NEC?) dealing with network grounding where it goes between buildings, but my IT manager didn't have it written down. I'm not usually into the business ethernet network, but I want to be sure I've got my ducks in a row before giving him an answer since he also happens to be the guy who can buy the Autocad software I asked for in this year's budget. Anyone know where I should look? NEC? UL? Or are we ok since the hubs are plugged into 120VAC from the same transformer? Thanks, Susan

[TimWilborne 108]

NEC 800.93 Cable Grounding. The metallic sheath of communications cables entering buildings shall be grounded as close as practicable to the point of entrance or shall be interrupted as close to the point of entrances as practicable by an insulation joint or equivalent device Then looks like 800.100 - 80.182 for more. All I saw on a quick browse of the NEC

[ssommers 20]

Thanks TW! That got me what I needed.

[TimWilborne 108]

Isn't that what I'm on your head for? To look out for you?

[BobLfoot 301]

Not exactly related to your question, but in your situation I'd use fiber not wire cable between the buildings and have a network closet in the plant building where the fiber entered an intelligent switch and then went out as fiber. Just an observation.

[ssommers 20]

Hi BobLFoot - The current ethernet network was done piecemeal thru the 80's & 90's - first the office, then the plant offices. When they put it together & it worked, that's all they cared about - no long-term thoughts of expansion. The current IT manager is also the Plant Controller (finance) and admits he doesn't know about the wiring aspect - thus he came to me. I know just a tad more about wiring & hubs & stuff, but not enough about ethernet as a whole. After cleaning up the office network, fiber to the plant is the next step along with getting more bandwidth on our Internet connection. He's getting quotes now so I guess I'm going to have to learn a lot about fiber really fast. Any idea where I can find some basic fiber optics info for starters? Thanks... You guys are the best!

[PdL 71]

Googling "fiber optic networks" is something to start with. Here's an online course on fiber optics, not so much on networking but quite extensive. Here's another one. And another one. Have fun

[BobLfoot 301]

PDL's links are good. In my personal experiences fiber if done right is a do it and forget about it thing. For that reason we alsways found a reputable fiber installation contractor, {read between line here was willing to warranty work for 1 to 5 years}, and paid them to lay and terminate fiber. That may have been the cowards way out, but worked for us.

[ssommers 20]

Thanks for the links PdL. After a bit of reading, I think I understand enough to ask the contractor the right questions to ensure he's going to do a quality job. And I'll definitely be looking for a long warranty period. Buying services from a good contractor when you don't know how to install something is not cowardice... It's self-preservation. Thanks again!

[paulengr 44]

NEC requires power conductors to be grounded and bonded. Ethernet itself doesn't use a ground/neutral so that is immaterial. Grounding serves two purposes. Looking at a single building, the neutral and ground bus bar are connected together at either the transformer or where the feeder terminates (if you are dealing with a feeder situation). The ground connects to a good, solid Earth ground...which means you pound in a ground stake starting with an 8 footer and keep adding sections if needed (I live in an area with sandy soils...good grounding is frequently an issue). The purpose of this connection is to partly to absorb lightning strikes. If you run ungrounded, your equipment will get hit and chances are something will be melted into trash during every lightning storm. But mostly the purpose of the ground is to maintain a relatively neutral coupling between the Earth and the electrical equipment. If you don't do this, you WILL still have a capacitive coupling and you may notice all kinds of screwy things going on. Some plants do run "ungrounded" systems. If you are in that situation, you need to take special precautions when setting everything up. The electrical guys will know if it is an ungrounded delta. Second, you have an entire circuit which is NOT really connected to anything but serves to connect together all your panels, conduits, etc. Basically anything metallic and exposed that has conductors inside it is bonded (connected together) back to the ground bar. The purpose of this system is so that if a conductor happens to get loose and touches a metallic part, it will provide a low resistance path back to the neutral side of the fuse or circuit breaker and trip the circuit. If you don't have everything bonded properly and a "hot" wire does get loose and someone touches the live part, you'll have an electrocution problem on your hands. Between buildings, there are a couple approaches to doing this properly. Many people mistakenly simply drive another ground stake in and tie everything together in the power circuit as if it is a feeder. This is poor practice because now at least in theory, you can form a "loop" where you have connected a system that has one voltage (Earth ground) to another one, forming a complete circuit through the Earth. If you are going to do this, then you need to have a transformer in between, even if it's an "isolation transformer" (transformer turns ratio is 1:1). That way you've electrically isolated both systems. You bond the conduit surrounding your underground cabling only on one side in this arrangement. The alternative is to simply treat the second building as an extension of the first building...so you bond everything. No isolation transformer. No ground stake. This is all NEC Code 101 type stuff. So in virtually any place in the U.S., you are required to do it this way, whether the previous IT guy just managed to plug some stuff in and string cables off a step ladder or not. Note that I'm not one to push pulling permits or getting inspections done. I find this is a lot of expense and trouble and usually that the inspectors miss 99% of the real issues while they are busy looking for missing screws, arguing about whether the conduit should have doglegged left or right, and searching for UL stickers on everything. The reason that it should be to Code (other than the fact that the electrical codes are designed to cover real, actual events that other people have paid for with their lives or damage) is that if you ever do have a problem which requires an insurance claim, the insurance company will deny the claim if they find everything wasn't up to Code. That's the power side. Now moving over to the networking side, Ethernet already has 1500V 1:1 isolation transformers in the NIC's. All signals are sent in "differential mode"...the NIC card compares the voltage on one wire to the voltage on another wire. The actual voltages don't matter...only the difference. And that isolation transformer means that the average voltage on the wires will be zero anyways. So effectively whether it was intended to be that way or not, Ethernet is "ungrounded" by design...it doesn't care what the voltage difference (if any) is in terms of your neutrals and grounds. The inductors and isolation transformers also put several 10's of dB of isolation on low frequency signals such as 50/60Hz electrical signals which could somehow be coupled to your communication wiring...so as long as you don't do something like trying to run the feeder to your FM radio station transmitter in the same conduit, you don't really have to worry about isolation in the conduit...still it's good practice to keep power lines and communication lines separate. One of the primary reasons is that the insulation in the Ethernet is not usually rated for the same voltage as your power lines should the unexpected happen such as the guy they just hired to run a fork truck around the plant... The last thing to consider when running from building to building with any communication cabling is that the darned things always seem to be lightning attractors. Here, there are three options. First, run lightning rods out high and big enough to create a protective zone for your communication cable. Sorry, I had to say it. I grew up on a farm in the midwest where virtually everybody had lightning rods on their houses. Second, run it in conduit or run it in ScTP at the very least (remember...bond on only one side with ScTP). This provides a hopefully lower resistance path back to ground compared to your communication cable. Third, put good quality surge protectors in the system. I'm not talking about power to your switches here. I'm talking about surge protectors directly on the Ethernet cable. Even this might not be enough but if you do both of these last two steps and make sure your power line grounds are good, I've never had issues in practice even in places that routinely get lightning strikes. As was previously mentioned, if you want to avoid the lightning strike issue altogether, you can always switch to fiber. It is so much less trouble. The only recommendation I can make here is to stay as far away from multimode fiber. It's cheaper initially (it's plastic instead of glass). The NIC's are slightly cheaper as well. But most of the cost of fiber is in the labor, not the cable. And after 10 years or so, the stuff seems to be degrading in my plant. And there are no speed limitations (or distance for that matter) with single mode fiber. That's the reason long distance phone companies use it to go 20-40 miles per run with multi-gigabit speeds with no sweat. And if you ever need to add more to your existing run, you can always use CWDM...you put a prism on each end of the cable and run transceivers with different colors of light, which don't interfere with each other. A relatively cheap way to multiplex. This is impossible with multimode fiber.

[ssommers 20]

There's lots & lots of stuff to process here so it may take a couple days to digest it. We do have a good single Earth ground for the complex. I went & looked at the underground conduit that takes the Ethernet cable from building 1 to 2 and it is bonded to the steel in building 1. Add to that the fact that the power for building 2 is transformed in building 1 and then goes through another underground conduit (again bonded to steel in bldg 1), I think I'm good to go on the power side. For lightning protection, the plant roof is the highest peak around here (we have cow fields all around for 5 miles) & does have a lightning rod attached to the complex ground rod. All power & communications cabling is underground & runs about 20 feet. So lightning exposure is also minimal. I definitely want to look into surge protectors. We have enough power outages due to power station problems around here that spikes coming in at restart have cost a computer or two. And top it all off with trying to get management to go to fiber optics. Thanks for all the great info!

[Ken Moore 23]

If you do go with fiber, pull a lot more strands than you currently need. The cost at this point will be minimal. We installed our fiber about 15 years ago, I argued for and won pulling 28 strand fiber. In the beginning we only used two strands, but now we are upto 12, with one damaged and not usable (might could repair, but we just switched to a spare). The increased usage is do to installing a gigabit backbone 18 months ago, we divided the site into 4 seperate networks, to increase band width. We are also leasing space to another company in on building, and they are using one pair also.

[paulengr 44]

Keep in mind what a "surge protector" usually is. It is usually a MOV (metal oxide varistor), but a much better device is a GDT (gas discharge tube). You can buy the GDT's directly from Newark, but Joslyn also makes a nice prepackaged version. In either case the way they function is that once the voltage exceeds a particular level, the device shorts the power lines to ground temporarily until voltages return to normal levels. The MOV does this as a voltage-variable resistor. The GDT does it by ionizing the gas inside into a plasma (nicely conductive). The advantage of a GDT (other than price) is that the reaction time is much faster and it doesn't have any "leakage current" to speak of. In other words, this stuff will save you from damaging surges/spikes, but does NOTHING to eliminate the "reboot" issue. It MIGHT mitigate some of it but nothing like what you are expecting. If you have spikes only, a better device is a "power conditioner". The usual suspects (APC, Tripplite) sell decent ones with warranties on them. This eliminates some noise and also the spikes, and doubles as a surge protector as well. If you also have sags (if you have surges from the power company, you also most likely have sags...but without a power quality meter, most people can't tell the difference), then get full blown UPS's. They are much more expensive any plan on replacing them every 1-3 years (the batteries don't last all that long), but they definitely eliminate all the problems. In my plant at the server room, we have something like a dozen rack mount servers, plus 3 or 4 Cisco 2500 series switches plus a couple 3550's, plus a voice gateway, router, firewall, etc. In short, a lot of stuff. In that environment, we have two 3KVA UPS's that are good for a whopping 8 minutes of backup power. I'd go buy a big $10K-$20K UPS but it's pointless...at that point, you may as well look at generators for the money. The generator we have kicks in within 30 seconds...the UPS's are just there to hold the network up in the interim. So at least in the core where the servers are at (I share the space with the IT guys but they like my power better for some reason), we have 24/7 availability and reboots are now only a software issue. It cost me close to $25K to get it to that point but it was well worth it.

[ssommers 20]

Oh, I know all about the sags & spikes - and we've got both. I'm the only EE here, so I get the whole ball of wax - machines & facilities. Power quality issues are ranking at the top of my list right now with both ERW and Induction welders having occasional problems. We already have generators, but management opted to go for a manual transfer switch for the office instead of the automatic ones we have in the plant for critical equipment. They're also buying UPSes that are good for about 10 minutes for the servers so they should have time to either nicely shut down or walk into the plant to flip the switch. I'll put the batteries on a 2 yr replacement cycle in the Preventative Maintenance system so no one has to remember it. Ken - I'll definitely make sure they pull at least 4x more fiber than what management thinks we need now when it gets installed. Extra cable is cheap, retrofitting isn't. Edited 23 Jan 2007 by ssommers

[GerryM 2]

I found these two statements to be hilarious, and typical. Why is mangement even deciding which switch to go with? Ya know what I mean!!!!

[ssommers 20]

Yep, yep, yep... Actually, the plant manager decided that the auto switch would cost too much. So the maintenance supervisor ordered a manual switch all by himself (at least he bought the same brand) and had the 3rd shift electrician install it without even asking me for a drawing. You can bet I'm keeping my hands out of that circuit! I trust the electrician on 3rd a lot, but that trust only goes so far when emergency power is concerned... I'm patiently waiting for the next daytime power failure to see how mad the company president gets when someone has to walk through the plant to turn on the office power.