Sign in to follow this  
Followers 0
Twigums

Tracking the master clock

14 posts in this topic

Posted (edited)

We are having issue on a machine running kinnetix 5500 drives where we randomly get clock Sync Faults. 

This particular machine is set up as a slave looking to another PLC(L36-ERM) on a different station of the same machine as the time master.  There are about a dozen other PLC's(totally different machines) on the same network and most are set up as masters for their own systems and to my knowledge they have never had this issue.

There are also dozens of other devices connected to this network, and then this network is part of the facility wide network.( I apologize if this is a little wordy)

We are pretty sure that the slave is finding a more accurate clock on the network somewhere and syncing to it causing the servos to fault.

We have tried making both the master, and the local clock of these 2 particular PLC's higher priority than the others to eliminate this with no success.

What I am trying to do is track what clock this particular slave is looking at as its master when the clock sync fault has occurred.  is there away to do this through ladder logic?  it seems like we should be able to get this information and save it so we can figure out who the culprit is.

Edited by Twigums

Share this post


Link to post
Share on other sites

You can use a GSV instruction to access the TIMESYNCHRONIZE object that might give you the info you need.

Share this post


Link to post
Share on other sites

Thank you Joe E.  This looks promising. 

I just cant seem to get the data type right.  The GSV returns data when I use a DINT[8], but they are seemingly random numbers.  I have tried to correlate the information, but I do not see any relations to the master clock data.  I have also tried to make a SINT[8](as explained in the instruction help) and store but all I get are 0's in every field.  Other data types are either invalid or again I will get all 0's.  Im not sure what I am missing here.

Share this post


Link to post
Share on other sites

Hmmmm....

It looks like the GrandMasterClockInfo returns 24 bytes in a structure. The clock identity is 8 SINTs while the rest of the members are INTs (2 bytes each), so you probably can't just use an array of a single data type. If Clock Identity is all you want and you don't care about any of the rest, you may be able to get away with a SINT[24] array. I would instead create a UDT containing a SINT[8] array and then 8 individual INTs:

TimeSync_GSV_UDT.PNG.a2786e6752c3503d6f3

 

Share this post


Link to post
Share on other sites
Just now, Joe E. said:

Hmmmm....

It looks like the GrandMasterClockInfo returns 24 bytes in a structure. The clock identity is 8 SINTs while the rest of the members are INTs (2 bytes each), so you probably can't just use an array of a single data type. If Clock Identity is all you want and you don't care about any of the rest, you may be able to get away with a SINT[24] array. I would instead create a UDT containing a SINT[8] array and then 8 individual INTs:

TimeSync_GSV_UDT.PNG.a2786e6752c3503d6f3

 

I tried this exact UDT structure and it gave me an invalid data type error.  I will keep messing around with it to see if I can find something it likes.

 

thanks again for all the assistance!

Share this post


Link to post
Share on other sites

Posted (edited)

Just for fun, have you tried a SINT[24] array?

ETA:

You said you got it to work with a DINT[8] array. That's 24 bytes. It's a kludge, but go back to that and use a COP instruction to copy the first 2 elements of the DINT array to a SINT array. That should get you meaningful results in the SINT array.

For the COP instruction, the source would be element [0] of the DINT array the GSV writes to. Destination would be element [0] of the SINT array. Length would be 8 (number of destination elements to copy). That will copy the first 2 DINT registers into the 8 SINT registers.

To get the rest of the data, use another COP instruction with an INT[8] array. The source will be element [2] of the DINT array, destination element [0] of the INT array, length is again 8.

I don't have access to a PLC I can use to test this, so it may not work.

Edited by Joe E.

Share this post


Link to post
Share on other sites

Definitely a better result, but not quite the info I was hoping for.  I am assuming Sint's 0-8 are the Clock Identity. Sint [10] is accuracy, Sint[14] is offset, sint[20] and [22] are the priorities.

Just not sure if I will be able to identify the specific piece of hardware with this.

Master Clock.PNG

Share this post


Link to post
Share on other sites

Some ideas:

- If you use a managed switch you could create a VLAN where only the PLC and the desired driver can comunicate: very easy.

- Or if you add a firewall, or a cheap PC acting like a firewall (like a raspberry Pi) you could filter the IP's and/or MAC addresses to allow traffic only between the PLC and the desired drives. Not so easy, but same result as above.

Cheers

1 person likes this

Share this post


Link to post
Share on other sites

Page 9 of this manual says the clock identity is usually based on the MAC address. Change the format of your SINT[24] tag from Decimal to Hex and see if you can correlate the values to the MAC ID of any of the PLCs. I did that with the values you show and then looked up the MAC id here. It says it's a Rockwell Automation device.

1 person likes this

Share this post


Link to post
Share on other sites

Posted (edited)

That did it Joe E.  This is exactly what I am looking for.

16 hours ago, empanada said:

Some ideas:

- If you use a managed switch you could create a VLAN where only the PLC and the desired driver can comunicate: very easy.

- Or if you add a firewall, or a cheap PC acting like a firewall (like a raspberry Pi) you could filter the IP's and/or MAC addresses to allow traffic only between the PLC and the desired drives. Not so easy, but same result as above.

Cheers

Unfortunately we do not have a managed switch on this machine. 

If we cannot remedy this with a simple change to the device that is taking over as the master, then going to install a managed switch and try to go that route.  For policy reasons, if we were to add a PC the I.T. Dept would have to be involved since it is connected to the Facility network :/.  That usually doesn't end well here.

Thank you both very much for all of the help on this!

 

Edited by Twigums
1 person likes this

Share this post


Link to post
Share on other sites

I hear you about involving the IT department. At my previous employer, we couldn't have anything other than a simple unmanaged switch without permission from corporate IT. And they would have full and sole control over it. Even a simple NAT device wasn't allowed. We probably could have gotten some to put in machines as long as they never (even accidentally) connected to the company network, but everywhere we considered doing it, connecting to the company network was one of the main goals.

Share this post


Link to post
Share on other sites

Posted (edited)

In my opinion the control network must be completely isolated from the IT network, and even from the Internet, if possible. The potential risks are enormous , not only economics, but, also for safety of the workers and even outside the factorys.

The connection between both worlds must be minimal and the risks must be measured carefully.

In a lot of cases there is no real need to have the control network exposed to those unnecesary risks, and someone must says to the CEO that view the factory on real time from his smartphone it's very cool on the work meetings...but in a number of cases, it's not more than a toy, a very, very risky toy. 

Cheers

Edited by empanada
1 person likes this

Share this post


Link to post
Share on other sites

Agreed!

Try telling that to these Jokers.  Data collection that no one even uses is more important than keeping your machines shielded from outside influences.

They do take security seriously, I am not trying to say any differently.  But the security also becomes the problem in some cases.

Also, when I.T. goes to messing around with network settings, and machines start doing funny stuff, it becomes our problem to troubleshoot and fix.

I will stop whining now... lol

Be Safe out there!

Share this post


Link to post
Share on other sites

I've been in a number of IT versus Plant Floor discussions in my 30+ years doing this.  Let me encourage you to start by establishing the Vocabulary.  There are two worlds IT and OT.  Information Technology and Operational Technology..  They use many of the same tools and hardware, but have drastically different paradigms to operate in.  Once you get IT buyoff that you're simply trying to run the OT world for the benefit of the company, I've found they get much more cooperative and less defensive.  You're not invading their space, just trying to optimize yours.  I say this all in about 30 seconds, but it's taken 5 years at my present company to reach an understanding that benefits all.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0