NTP Reflections

First, if you will indulge us briefly, we must thank the RIPE NCC for approaching us to comment and contribute a piece here at RIPE Labs. While this author has not had the pleasure of visiting the RIPE NCC or the opportunity to attend a RIPE Meeting in person, the service the RIPE NCC provides not only to its own community, but what it makes publicly available to the world are just some of its output we have admired from afar for some time.

Before we are accused of pushing this flattery too far, let us begin. Ringing in the new year, a number of discussion forums lit up with talk of widespread denial of service attacks involving the Network Time Protocol (NTP). Most of you will undoubtedly be aware of DNS-based amplification and reflection attacks, a threat that rose to prominence nearly a decade ago. It turns out, to the surprise of almost no one, DNS is not the only means by which this particular class of attacks can be propagated. That is, there exists many ways with which source address spoofed requests can induce multiple or large responses from widely available listeners. A concentrated amplified response can often overwhelm the average host whose address was spoofed, and sometimes even severely or terminally degrade service for the not so average host and network.

The NTP happens to be the latest service used in a variety of amplification and reflection attacks that has sparked the attention of many operators. Rather than simply review the NTP attacks and how they are conducted, we would instead like to present a narrative, a timeline of the threat we saw coming. For once perhaps, these NTP attacks were anticipated since at least 2009. The story of what the community, or some part of the community, knew, researched and did leading up to the "in-the-wild" attacks we believe is worth considering. Of course we must admit that this is one perspective on events leading up to the attacks and our's may be an imperfect one. We can only relay events as best we have experienced and recall them. It seems worth recording
that experience for posterity, but also to stimulate discussion.

By 1992, NTP was already on its third revision and currently it is in its fourth, which was only recently published as IETF RFC 5905 in 2010. The length of time between NTPv2 and NTPv3 was far shorter, but perhaps it may surprise you to know that while NTPv4 is already widely deployed you can still find plenty of devices that are using version 2 and 3 of the protocol. This leads us to what ought to be a recurring theme in issues such as these in terms we'll borrow from Rich Seifert, "Never underestimate the installed base." Perhaps fittingly, David Mills , the now mostly retired Internet luminary responsible for NTP's development was awarded the IEEE Internet Award in 2013 for his work on Internet time synchronization.

History of relevant NTP research

Our story of the NTP attacks goes almost as far back as NTPv3 itself. In 1994, USENIX published a paper titled Experiences with a Survey Tool for Discovering Network Time Protocol Servers by James D. Guyton and Michael F. Schwartz both affiliated with the University of Colorado, Boulder at the time. The authors discuss their experience using the monitoring capabilities of accessible NTP systems. This "monitor" mode is a great debugging tool, but it can also be used to surreptitiously uncover relationships and, as recent events attest, as a packet amplification vector. These authors were only concerned with surveying NTP system relationships for legitimate measurement purposes and if aware of the amplification threat, they made no mention of it.

Similar studies using what in some NTP software is simply known as the "monlist" command continued to appear. In 1999, Nelson Minar then of MIT released his results in a paper entitled A Survey of the NTP Network . Later still was a NTP survey conducted in 2005 by Cristina Duarte Murta and Pedro R. Torres Junior at the Federal University of Parana in Brazil.

Noticeably missing from any of these early NTP surveys was any serious discussion about the potential for abuse yet the amplification and reflection threat has been present since 1994. It was only recently that we began to see attacks come to fruition. Was anyone aware of the threat before December 2013?

On March 2, 2010, HD Moore, a well known figure in the security community for having developed the Metasploit penetration testing platform, gave a talk at BSides San Francisco . In the second half of his talk, Moore summarizes his recent activity exploring NTP and public NTP systems. Moore shows his audience the wealth of survey-related information that can be gleaned from NTP, very much akin to published reports from years earlier. Shortly after this talk, Moore and others later highlight the DDoS potential certain NTP implementations and features may exhibit though at the time the issue does not gain much traction beyond a typical security news cycle.

Developing a Secure NTP Template

When we first heard about Moore's recent NTP interest, we took notice. In early August of 2009 we had released a draft Secure NTP Template web page to a small set of private security communities where Moore is not present. In the first week of August we had independently rediscovered NTP as a rich source of survey data, but also as a potential DDoS threat, the latter a finding we appear to have been the first to realize. This led us to review some of the earlier survey efforts mentioned above and when we quietly published our secure NTP template later in December of that year, we mentioned the potential for DDoS attacks, but did not bring any particular attention to it.

In those same security communities we originally floated the draft version of our template, we also announced we would be regularly looking for open NTP servers with the intent to feed our results back to operators. So by the time Moore gave his talk, and actually it was a few weeks later when we heard about it, we were well on our way in thinking we were really ahead of the curve on this potential problem. Exacerbating the issue, it was just one day after we heard about Moore's efforts, that we saw a change from the developers of ntpd, the reference implementation of the protocol, to change the monitor list function in a way that would raise the maximum amplification factor even higher.[1] We realized immediately it was time for an intervention!

So with millions of NTP systems discovered, a template already published, some trusted folks already aware of what we were doing and the worry that it might be a very short amount of time between the Metasploit NTP plugin and miscreants in the underground adapting it, we contacted the ntpd developer who had posted the monlist change, alerting him and the core ntpd development team to our concerns. By the end of the month, the ntpd developers had made changes to the code base and were on a path that would greatly enhance the default security posture of ntpd in the future.[2][3] If Moore ever published a paper on the NTP DDoS threat as he once suggested, we never saw it. For awhile, everything seemed quiet and we appeared to have dodged a bullet.

The first NTP attacks

Then in late 2011, some NTP operators were reporting what looked like the NTP amplification attacks we knew were possible.[4] While the ntpd developers and others like us paying closer attention noticed the discussion, these reports otherwise seemed to go largely unnoticed. As soon as they came, away they went. We didn't see or hear of any more NTP DDoS attacks. Not for at least a couple of years. Again, we thought we were going to quietly allow the natural progression of software to mitigate this threat out of existence.

Then in November 2013, after many people seemed to have forgotten the issue, some NTP operators again reported seeing amplification attacks.[5] Discussion of these attacks again took place on the same NTP-oriented operator list as in 2011. This time, the topic was then brought to the attention of folks participating in a private operational security forum where we participate and had originally published our secure NTP template along with disclosing our probing activity. In response to one of our follow up messages summarizing the status of the NTP threat and our work on it, one of our colleagues followed up with us offline to review an independent investigation and survey that covered in part this threat.

Our colleague was preparing a public blog posting describing the NTP threat in very specific detail and planning for it to be linked to from CERT (computer emergency response team) alerts. The brief was accurate and well written, but we feared it would only draw even more unwanted attention of an otherwise still little known issue. In the discussion that spawned this offline exchange we had estimated it would be at least another couple of years before most Unix distributions started running a fixed version of ntpd. We were hoping, foolishly it now seems, to be able to get away without making hay out of the issue for awhile.

Recent NTP attacks

It was only a little more than a month later, in late December of 2013 when NTP-based attacks reared their ugly head again and this time lots of operators either felt it or took notice. Game on as they say.

Almost immediately, network and security operators rushed to the scene, some to offer guidance based on past work, such as how to lock down NTP configurations or with suggestions to upgrade to the latest ntpd code. The miscreants were clearly building lists of NTP reflectors so too were a few well-meaning colleagues and security organizations including one by our good friend Jared Mauch who makes much of the data more or less publicly available.[6]

While we had been prepared and gathering data for some time, unfortunately we were caught with our Internet pants partially down. While we had been continuing to collect data about open NTP systems, we hadn't been making this available as a regular feed to our partners. It was supposed to be available, but at some point, none of us are sure why, "it was turned off". So we scrambled to get that going again and manually send data to people who wanted it.

As we began feeding data about open NTP systems to people again, and as our secure NTP templates were passed around various mailing lists by those who knew about them, we began to ramp up and provide whatever support we could. We were given suggestions for improvements to the secure NTP template and of course we were invited by the RIPE NCC to talk about the issue here. What seems like an eternity in the security field, our NTP data was always there, literally going on two years. Perhaps it isn't until we are all forced to confront a really good reason to to use it, interest cannot be easily stirred while other, more pressing problems occupy our day.

One unexpected bit of feedback we received, not aimed specifically at us, was on the increase of Internet-wide probing and the side effects of this traffic for some segment of Internet users. While Internet surveys are both interesting and potentially very useful, operationally they do present certain challenges. For instance, a large ISP who has low rate mobile, near-line clients, may have to face complaints of fees that are activated when these near-line clients go active for even the smallest amount of traffic sent or received, including traffic that the client may wish to avoid if possible. While there are ways for an ISP or its customer to mitigate unwanted traffic, should they expect, by default, communications silence from the rest of the Internet? This pits legitimate survey folks up against not so legitimate miscreants who are looking to take advantage.

Conclusions and open questions

After the December 2013 NTP attack wave, perhaps the bigger questions reflect what would have or could have been different. Did any of our early involvement in the matter help?  If so, how much? Dealing with hundreds of thousands of open NTP servers in a variety of products and networks seems nearly impossible. Our experience with open DNS resolvers hasn't been very encouraging. Do these sorts of ubiquitous threats only really start to go away when all old vulnerable code is retired, defaults are changed in the majority of systems or if most ISPs agree to enable or disable some capability in unison? We note that this last idea rarely if ever seems to occur. If we had more space, here too we might say something more about underestimating the installed base.

While we can hope to get better about interpreting the signals we see as threats mature, what we will probably never be able to know with certainty is the narrative of the NTP attacks from the opposite side of the table. Who initiated the early attacks? How did they discover them? How did that knowledge make its way through the underground and begin to promulgate?

In hindsight, when the attacks reappeared in November 2013 and when we heard about others working on the issue for the security community at that time, we should have been better prepared with data and should have encouraged again, perhaps more forcefully and openly, that operators start taking the issue seriously and do something about it. Yet, we've now had about a month to do that and while more people are aware of the issue and many NTP systems have changed, the total number of potential NTP amplifiers hasn't changed all that much. Are we relegated to wait for the newest ntpd code to matriculate throughout the net? What more can we do?

The one thing we can be sure of, is that someone, if not us, will have an opportunity to relive a situation very similar to this with the next threat. Will there be a next threat? Yes. How will we, we being you, us, everyone reading this, react to it if and when we find it first? How should we react?

[1] ntp-dev 4.2.7p22 MRU list cap raised from 600

[2] Require nonce with mrulist requests

[3] remove ntpd support for ntpdc's monlist (use ntpq's mrulist)
 
[4] Odd surge in traffic today
 
[5] DDOS using my ntp server
 
[6] OpenNTPProject.org - NTP Scanning Project