Precision Time Protocol (PTP) allows many devices on the same network to maintain extremely precise times in relation to each other. However, different time standards and a device's interpretation of timing signals can cause issues.
There are two time standards widely used across the globe, International Atomic Time (TAI) and Coordinated Universal Time (UTC). TAI is based on a high-precision atomic time using an average of 450 atomic clocks worldwide. However, most clocks do not use TAI time, rather using UTC which is widely used in most cases. Due to the slowing rotation of the Earth, a number of leap seconds differentiate the two time standards. Currently the offset between UTC and TAI is 37 seconds, meaning UTC is exactly 37 seconds behind TAI. This offset is known as leap seconds and can change periodically twice a year. The last leap second was added in 2017. For a deeper dive into these two time standards refer to our blog on UTC, TAI, and Leap Seconds.
Because two time standards are used, syncing many devices simultaneously can cause headaches when developing time-critical systems. There are many different variables which can affect the timing coordination between devices including PTP grandmaster priority, network switch clock configuration, and device's time sync resources. There is a unique combination of all these variables that will work for any particular system, but hopefully these troubleshooting steps will provide some context and help when a rogue UTC offset is added or PTP grandmasters are wrong.
Hardware
The solutions in this article were derived using the following hardware:
- cRIO 9049
- PXI 1095 Chassis
- PXI 8881 Controller
- PXI 6683H Timing Module
- Cisco IE5000 Network Switch
- Microchip Sync Server S650
Issue
In an attempt to time synchronize the cRIOs and the PXIs on the network to the Microchip Sync Server (PTP Grandmaster), the PXI 6683H card was applying the UTC offset making the two device type's time exactly 37 seconds apart while both device types indicated being synced to the Microchip Sync Server.
Troubleshooting
Grandmaster Priority
The first step to troubleshooting any time sync issue is to ensure the desired time grandmaster, often the device sending out PTP packets, has the highest priority. A higher priority device has a lower priority number and network switches use the Best Master Clock Algorithm (BMCA). The possible priority numbers range from 0 to 255. Setting the grandmasters PTP priority to the lowest priority number on the network will ensure that the PTP packets forwarded to the devices are coming from the correct device.
Some sync servers have the capability to see which devices are synced to it which can aid in determining if the devices are synced to the sync server. This is not always accurate however as the PXIs and cRIOs in this particular hardware setup indicated they were receiving their time from the Microchip grandmaster, but the Microchip grandmaster indicated that no devices were synced to it. While this could be cause for concern, this lack of indication on the grandmaster’s side was never a root cause.
If a device indicates it is time synced to a device other than the dedicated grandmaster, check the PTP priority numbers on each device on the network and update accordingly. It is best to have a standard “not grandmaster” priority number far higher than the dedicated grandmaster’s priority number.
Device UTC Offset
PXI
PXI Chassis and controllers do not have an internal timing module and require a separate timing module card to participate in time synchronization. Due to the timing module, the PXI requires a few extra steps in order to configure a Time Sync Resource. For more information on configuration refer to DMC’s Time Sync Configuration article.
Because a PXI Chassis uses a timing module, NI MAX test panels make it very simple to review the current time sync resource settings and UTC offset. To access the test panel, open NI MAX and navigate to the timing card of the desired device. Select Network Devices, then the desired PXI chassis, and then the 6683H card slow (or whatever timing card is installed). Open the Test Panel for the timing card. The clock type and current UTC offset will be displayed as follows. If the UTC offset is different than expected, there are two options to solve it.
- Manually update the offset to the desired number of seconds to sync with the rest of the devices.
- Programmatically set the offset when initializing the time sync resource in LabView. Below is an example of how to set the offset programmatically. For more information on time sync with PXI-6683(H) modules refer to our dedicated blog IEEE 1588 Synchronization using PXI-6683(H).
For more information on time sync with PXI-6683(H) modules refer to our dedicated blog IEEE 1588 Synchronization using PXI-6683(H).
cRIO
cRIO devices have an internal timing module and do not require an extra card, so viewing the time sync resource must be done programmatically and stored in a log. For more information on configuration refer to DMC’s Time Sync Configuration article.
NOTE: In this hardware configuration, the cRIO never applied an undesired UTC offset so testing/troubleshooting noted here is possibly limited.
Network Switch PTP Clock
Depending on the network configuration and the requirements of the RT devices, it is possible that using a different PTP clock configuration on the network switch could resolve the issues. This article will not explain the different PTP clock types and their messaging, but rather explain the differences observed when using a Transparent Clock vs a Boundary Clock.
Transparent Clock
This system was originally configured to use a Transparent Clock, which is where the rogue UTC offset was first observed. Updating the offset in the LabView code programmatically allowed the PXI to sync to the grandmaster with the correct time.
However, due to system requirements the PXI timing cards needed to be on a separate VLAN. Unfortunately, the Cisco IE5000 cannot send a PTP signal using the Transparent Clock configuration when multiple VLANs are configured on the switch.
Boundary Clock
Unlike the Transparent Clock configuration, Boundary Clock allows multiple VLANs and sends the PTP signal to all VLANs. This allows for system synchronization without the need for extra sync servers, or extra PTP ports configured on the sync server. Upon reconfiguring the network switches to use a Boundary Clock, the PXI time no longer applied the rogue UTC offset and the programmatic offset updates were no longer required.
It is still unclear why the PXI 6683H card applied the offset on the Transparent Clock configuration and not on the Boundary Clock configuration. To read more about DMC’s findings and assumptions on PTP Clock Type refer to DMC’s Time Synchronization and PTP Clock Modes article.
Conclusion
Time synchronization is an important tool for many systems, and it requires both accuracy and precision when collecting data at high speeds. Leap seconds and the UTC offset from TAI can plague systems. An untimely jump 37 seconds in the future or in the past can cause critical failures or data loss, so be sure to check each device's time and protocols to ensure unwanted behavior is not observed.
Learn more about DMC's Test and Measurement Automation expertise and contact us for your next project.