Increased Query Load on Root Name Servers

Wolfgang Nagele — Jun 29, 2011 03:15 PM
Filed under: ,
We experience an increased query load on K-root, the root server operated by the RIPE NCC. See below some analysis and graphs showing the traffic on K-root and other root servers.

Since 28 June at approximately 17:00 (UTC), we observe an increased number of queries on K-root and other root servers. In Figure 1 below, you can see the number of queries per second seen at all K-root global nodes.

Increased queries on k-root - global nodes

Figure 1: Number of queries per second seen at all K-root global nodes since yesterday

We see around 40,000 - 50,000 queries per second more than usual which is about four times the normal traffic load on the system.

You can see traffic shifting from AMS-IX (Amsterdam) to LINX (London) shortly before 20:00 hours UTC. We intiated this shift in order to gather data about the traffic source.

We are not yet sure what causes this increased query load, but we are in contact with ISPs that are carrying this traffic in order to investigate further. The root name server operators are coordinating observations and keeping a close eye on the developments. We update once we have more information.

Below you can see a series of DNSMON graphs that show how this incident affected the responsiveness of each of the root name servers. 

root from 61 Probes (AVERAGE)Figure 2: IPv4 unanswered queries for domain 'root' from 60 probes (Average)

 

  • Yellow = less than 66% of queries on average do not receive a response
  • Orange = more than 66% and less than 90% of all queries do not receive a response 
  • Red = more than 90% of the queries do not receive a response
 
The following graph shows the individual measurements for K-root during the same period.
You will notice that only a few test clients were affected because not all anycast instances of K-root were subject to the increase in traffic. Also not that we successfully mitigated the effect of the increased load even further around 06:30 UTC:
 

k-root 29 June 2011Figure 3: K-root server

 

  • Horizontal = only single measurement nodes saw a problem
  • Vertical = most of the measurement nodes saw a problem
This overview of current DNSMON measurements for the root name servers is publicly available. For more explanation of the DNSMON service, please refer to http://dnsmon.ripe.net

It is important to note that these measurements show that the root zone is fully available even though some root name servers show degraded performance. We expect no noticeable degradation of any Internet services caused by this.


 

9 Comments

Ian Meikle
Ian Meikle says:
Jun 29, 2011 03:31 PM
Did you manage to characterise the traffic? I have seen jumps in traffic caused by MX queries which generated large amounts of NXDOMAIN responses previously, though these were not as long lasting.
wnagele
wnagele says:
Jun 29, 2011 03:38 PM
So far they have been queries for QTYPE=A.
Daniele Sluijters
Daniele Sluijters says:
Jun 29, 2011 08:55 PM
Anything new / interesting to note?
Wolfgang Nagele
Wolfgang Nagele says:
Jun 30, 2011 10:22 AM
K-root is not having problems in coping with the additional amount of traffic. Our efforts are currently focused on back-tracing the possibly spoofed sources with upstream providers.
anandb
anandb says:
Jun 30, 2011 12:33 PM
If you're wondering why there are no statistics for the K-root instance in London, it's because we've paused the processing of the stats. We're still collecting them, but not post-processing them, in order to divert all capacity to answering queries. Eventually, all the staticstics will be visible.
Wolfgang Nagele
Wolfgang Nagele says:
Jun 30, 2011 04:00 PM
We saw this traffic drop to about half the volume it used to be at it's peak.
Daniel Karrenberg
Daniel Karrenberg says:
Jul 01, 2011 11:12 AM
Just a quick update: The number of the "unusual queries" has decreased. We are working to further mitigate the impact by distributing them more evenly among the anycast instances of K-root. We are busy analysing the characteristics of the "unusual queries". So far we do not have publishable conclusions and we do not expect to have them this week; we will of course publish our conclusions once our analysis is solid enough.
Brett Carr
Brett Carr says:
Jul 02, 2011 11:47 PM
We have seen similar (and larger) jumps in traffic in recent months and also observed the fact hat RD=1 on all the queries, we didn't find any pattern in source addresses though. Would be ver interested in anything you find.
Wolfgang Nagele
Wolfgang Nagele says:
Jul 04, 2011 10:03 AM
Hi Brett,

We are currently starting our analysis of the events and digging deep into the traffic details. We will share our information as soon as we are finished with this analysis.

Regards,
Wolfgang
Add comment

You can add a comment by filling out the form below. Only plain text is possible. Web and email addresses will be transformed into clickable links. Comments are moderated so they won't appear immediately.

Related Items
Visualising Bandwidth Capacity and Network Activity in RIPEstat Using M-Lab Data

As a result of the cooperation between the RIPE NCC and Measurement Lab (M-Lab), you can now ...

RIPEstat 2013 Year in Review

RIPEstat users saw a lot of changes throughout 2013, from support for new query types, such as ...

Internet Disruptions in Sudan

Significant Internet disruptions are happening in Sudan, possibly as a reaction to riots. We use ...

IPv6 RIPEness - Implementing the Fifth Star

In this article we present the first publicly available beta version of the fifth IPv6 RIPEness ...

Root Servers in Member Networks

In addition to the existing global and local instances of K-root, we propose member instances of ...

more ...