On 31 December this year, we're scheduled for another leap second. There are many stories about what leap seconds can do to infrastructure and applications, and rituals are built up around them. Such rituals stem from reality: leap seconds trigger poorly-tested code paths and run contrary to assumptions that system time always runs in one direction. It's useful to be aware of how your infrastructure handles leap seconds and how NTP servers handle them, so you can plan around the event. Here, we look at some of the NTP measurements the RIPE Atlas platform took around the last leap second, and approaches for handling them.
Background
The ITU standard on time signals specifies that leap seconds can be inserted or removed at the end of any month; we do this to keep Coordinated Universal Time (UTC) within one second of astronomical time. Historically, we've only ever inserted leap seconds, and even then only at the end of the months of June or December. We're fast-approaching the insertion of a leap second at the end of December 2016.
We can't programatically determine when leap seconds will happen, because the insertion or removal of a leap second relies on measurements of the rotation of the Earth. Leap seconds are announced months ahead of their occurrence by the International Earth Rotation and Reference Systems Service (IERS). Fortunately, we typically use the Network Time Protocol (NTP) to keep clocks close to UTC, and NTP can also notify hosts about these leap seconds.
NTP's approach to handling leap-seconds is two-fold: first, the NTP protocol specifies a Leap Indicator (LI) flag which informs clients that a leap second will be inserted or removed at the end of the current month (RFC 5905, section 7.3). As for the NTP clock itself, NTP's approach to inserting a leap second is to halt the clock for the duration of the leap. In doing so, the NTP clock never steps backwards.
Note that NTP specification for handling leap seconds is different to the Portable Operating System Interface (POSIX) specification. To insert a leap second, POSIX jumps the clock back to effectively replay a second, and so host-level timestamps can be ambiguous.
Measuring 2015-06-30-23:59:60
We have two long-running RIPE Atlas measurements that look at the NTP pool:
These measurements captured the last leap second in June 2015, so we can take a look at what we measured at the time. You can read more about our long-running NTP measurements on RIPE Labs: NTP Measurements with RIPE Atlas (by Philip Homburg). Note that these measurements are separate from any actual clock adjustments running on the probes.
How NTP Pool Servers Handled the 2015 Leap Second
RIPE Atlas probes for these measurements will resolve the pool name for every sample they take, but the NTP pool employs DNS load balancing to spread requests across the pool. This means that individual samples from the same probe in this data are likely to query distinct pool servers. It also means that we get broad coverage across the NTP pool so we can see deviations in server behaviour, but that an individual server might not be measured by the same probe more than once. Instead, it might be measured by multiple probes at different times, or only appear in the data once.
The LI flag can be set by NTP servers when they are configured with, say, the location of an authoritative leap second file. But given that the pool servers are provided on a volunteer basis, configurations vary. The following graph splits NTP servers into their strata (the number of NTP server hops from an authoritative time source), and plots the percentage of unique NTP servers measured that are setting the LI flag per-hour:
This mirrors work published in The Leap Second Behaviour of NTP Servers (by David Malone at the Traffic Monitoring and Analysis Workshop, 2016). The interesting high-level points are that:
- The leap indicator is commonly set 24 hours before the leap second actually occurs, increasing slightly towards the end of the day
- The leap indicator is reset in most cases on or around the leap event, so as to not advertise a leap second at the end of July
- A number of servers set the leap indicator after the leap second
- Finally, the leap indicator is never set by some servers
The data for subsequent months indicates that the LI flag is set in around 1.9% of the NTP pool servers observed at the end of July, around 1.5% at the end of August, 0.7% at the end of September, and 0.3% at the end of October. They may be configuration errors or bugs, but they're all erroneous.
What RIPE Atlas Anchors Saw
The NTP protocol doesn't prescribe what a system must do if NTP observes an upcoming leap second. In the graph below, I've pulled out a subset of RIPE Atlas anchors, which generally all have well-synchronised clocks. The anchors run openntpd, an NTP daemon which simply ignores the LI flag and therefore doesn't instruct the host system about the leap second. Openntpd instead gradually adjusts the host's time towards the time indicated by NTP. In doing so, its codebase is much simpler and no POSIX-style jumps happen, but systems will be up to a second out of sync for some time.
The datapoints provided by the measurements allow us to observe the relative offset between the local time on the anchor, and the time indicated by the NTP servers. Given these anchors aren't configured to step time backwards in the event of a leap second, we can see the leap second happen as follows:
Each individual line is the offset between an anchor's local time, and what NTP reports. In these cases, the most obvious takeaway is:
- The servers that the anchors are measuring inserted the leap second at the end of June as expected, and the offset reported immediately after that point in time is one full second
- In general, it takes these hosts about 18 hours to reach NTP time after the leap second
- In some cases, the algorithm over-compensates and adjusts too far in the wrong direction before stepping back.
In this graph, I've looked specifically at anchors that are measuring NTP servers that set the LI flag. In the case of the NTP servers that never set the LI flag, they also don't appear to insert a leap-second, which implies there's a disparity between NTP servers around these events. The NTP algorithms as defined should let clients ignore the outliers, presuming clients don't happen to query a majority of misconfigured servers.
Approaches to Work With, or Around, the Leap Second
Many deployments can allow the leap second to happen without issues, but it's wise to know for sure that your systems and applications won't break when the leap second occurs. In some cases stepping clocks back in time is not desirable, and in other cases the leap second has triggered kernel bugs. A couple of broad alternative approaches include:
- Increasingly common is "smearing" the leap second across a period of hours. Rather than set the leap indicator and pause the NTP clock/step POSIX clocks back, this process elongates each second very slightly around the event to eventually arrive at the corrected time. Google does this on their NTP servers; Akamai will do similar on their infrastructure, and Amazon also runs a similar algorithm. Microsoft Azure has in the past adopted a different approach with its time servers. These are all distinct from how the majority of NTP pool servers behave.
- Some organisations choose to play it safe by avoiding the leap second entirely, disabling ntpd some time before the leap indicator flag is observed to avoid the system stepping the clock back, instead reenabling ntpd after the event and allowing ntpd to skew clocks into sync.
In many cases, of course, you're likely to manage your own infrastructure or use your provider's infrastructure. It's wise to understand how your time servers will behave, and also how your hosts will respond to the information their get from NTP.
What approaches are you taking to handle the leap second? How are you planning to make sure you have an uneventful New Year's Eve?
Measuring 2016-12-31-23:59:60
The long-running measurements above will keep running across this year's leap second. We'll also set up measurements against specific NTP pool servers to get a better picture of how they behave, and we'll also start measuring Google's public NTP server to watch their smearing process.
What other common sources for NTP time are you measuring, or should we measure? By all means share your RIPE Atlas measurements, or ask us to look at specific services! We'll write a follow-up post in early 2017 if we spot anything interesting in the upcoming leap-second.
Further Reading
- NTP Measurements with RIPE Atlas, Philip Homburg, RIPE Labs, 2015
- The Leap Second Behaviour of NTP Servers, David Malone, Traffic Monitoring and Analysis Workshop, 2016.
- Rot at the Roots? Examining Public Timing Infrastructure, Kanthaiah Vijayalayan and Darryl Veitch, IEEE INFOCOM, 2016
- Leaving it to the Last Second, Geoff Huston's article covering time systems, and issues surrounding leap seconds.
Comments 1
Comments are disabled on articles published more than a year ago. If you'd like to inform us of any issues, please reach out to us via the contact form here.
randy bush •
excellent article. as this is an ops-oriented community, it may be worth pointing out that bsd and linux kernels are believed to handle the leap second correctly. but if you have devices such as ntp strat 1 devices, or other devices which are sensitive to the leap second, it may be worth investigating how the device handles it. e.g., i have a Time Machines TM1000A GPS Network Time Server. search for 'leap' in the manual fails. and i have yet to find anything that sounds real in the net of a thousand lies. but darned good article.