The Many Instances of the L-root Name Server

Stéphane Bortzmeyer — Jan 20, 2014 04:50 PM
The techniques used by the L-root DNS server are documented in RFC 7108 "A Summary of Various Mechanisms Deployed at L-Root for the Identification of Anycast Nodes". These techniques can be used by the RIPE Atlas probes to find the instance of L-root they are talking to.

Introduction

RFC 7108 is available online. It documents the various techniques that a user can try to find out which of the instances of the anycast cloud he/she is using. Some are traditional (CH/TXT request of id.server), some more modern (NSID) and some are specific to L-root such as the name identity.l.root-servers.org that you can resolve on your machine and that returns an IP address that is unique per instance. This last technique can be used by ordinary people, who have no access to the dig command or similar programs.

Methodology

Let's first use the NSID queries to examine the many instances of L-root. We already used this technique in another article and the code is still on Github. Here, we will ask 500 RIPE Atlas probes to query the SOA of the root domain to l.root-servers.net. The worldwide measurement is #1071935. It is here analysed by another program.

Among the 452 probes which got a proper NSID response, we find 109 different instances of L-root. According to RFC 7108, there were 273 different nodes (and therefore identifiers) in November 2013. We can see that the selected RIPE Atlas probes are not dispersed enough to find all the nodes. Maybe a test with more probes would help but one measurement is currently limited to 500 probes.

Results

The most popular instances are:

Root Server InstanceNumber of ProbesPercentage
ods01.l.root-servers.org 39 9%
kbp01.l.root-servers.org 28 6%
ory01.l.root-servers.org 19 4%
lba01.l.root-servers.org 16 4%
ham01.l.root-servers.org 13 3%
sjc01.l.root-servers.org 11 2%
ams01.l.root-servers.org 11 2%
her01.l.root-servers.org 10 2%
atl01.l.root-servers.org 10 2%
mmx01.l.root-servers.org 10 2%

 "Most popular" is to be taken with a grain of salt: RIPE Atlas probes are not evenly geographically distributed.

The instances are named after the IATA airport codes. "ods" is Odessa in the Ukraine, "kbp", Kiev, also in the Ukraine, "ory" is Paris in France etc. The popularity of Ukrainian destinations is a bit surprising. It is stable (the same measure two weeks ago or one month after showed the same phenomenon.) Of course, if we measure from a different region, we get a very different picture. Limiting to probes to the "West" (the US) area, we get (measurement #1071936):

Root Server InstanceNumber of ProbesPercentage
sjc01.l.root-servers.org 46 10 %
atl01.l.root-servers.org 43 9 %
bos61.l.root-servers.org 34 7 %
cjr01.l.root-servers.org 34 7 %
mvd01.l.root-servers.org 27 6 %
udi01.l.root-servers.org 22 5 %
yyz01.l.root-servers.org 21 4 %
phl01.l.root-servers.org 16 3 %
lax15.l.root-servers.org 16 3 %

Here, US cities (sjc = San Jose, atl = Atlanta, bos = Boston) get the lion share.

This change of "popularity" appears also when you measure in the South-East region (Asia-Pacific, measurement #1071938): a machine in Noumea (New Caledonia) is the most commonly used by the probes, before a machine in Perth (Australia). In North-Central (Europe, measurement #1071939), the Ukrainian sites are again in front. In North-East (Russia, measurement #1071937), we find of course the same results (but the number of probes is lower, so may be less representative).

All these measurements were done with IPv4. Testing with IPv6 show similar resultats, including the "Ukrainian attractor" (world-wide measurement #1402075).

RFC 7108 proposes another way, besides NSID, to find out to which instance you are talking to. You request the A record for the name identity.l.root-servers.org and then you get an IPv4 address which uniquely identify the instance. Do note that, for a given probe,  it may be a different instance than the one obtain by the previous measure. In the previous measure, we used a direct connection from the probe to l.root-servers.net, but now, the probe queries its default resolver, which may be far away, and may use a different instance of L-root. So, let's see the results of measurement #1071473, 500 probes requested:

IP addressNumber of occurrences
93.178.205.146 33
77.88.206.134 26
74.80.98.64 22
74.80.98.65 20
109.239.109.173 17

Many of these IP addresses have a PTR record which gives us an indication of the localisation of the instance. For instance, 93.178.205.146 is hostmaster-gw1.odessa.ucomline.net. So, the Ukrainian effect was a real one. There are few RIPE Atlas probes in the Ukraine. Can it be an attraction entirely caused by BGP? Or a misconfiguration? Here is an example seen from Free/Proxad (AS 12322) in Paris (France). NSID returns kbp01.l.root-servers.org and traceroute:

% traceroute l.root-servers.net
traceroute to l.root-servers.net (199.7.83.42), 30 hops max, 60 byte packets
 1  192.168.2.254 (192.168.2.254)  0.568 ms  0.426 ms  0.370 ms
 2  88.189.152.254 (88.189.152.254)  6.682 ms  6.768 ms  7.904 ms
 3  78.254.1.62 (78.254.1.62)  13.724 ms  13.741 ms  13.804 ms
 4  rke75-1-v900.intf.nra.proxad.net (78.254.255.42)  10.622 ms  11.369 ms  11.379 ms
 5  cev75-1-v902.intf.nra.proxad.net (78.254.255.46)  11.683 ms  12.649 ms  13.141 ms
 6  p16-6k-1-po12.intf.nra.proxad.net (78.254.255.50)  13.150 ms  10.321 ms  10.265 ms
 7  bzn-crs16-1-be1024.intf.routers.proxad.net (212.27.56.149)  12.052 ms  7.389 ms  11.070 ms
 8  th2-6k-3-po21.intf.routers.proxad.net (212.27.56.2)  26.276 ms  26.311 ms *
 9  jaguar-pni.routers.proxad.net (212.27.40.122)  10.974 ms  11.412 ms  11.901 ms
10  l.root-servers.net (199.7.83.42)  22.967 ms  23.503 ms  24.502 ms

The route shows only French operators and the TTL seem consistent with a route staying in France (typical TTLs for Ukrainian targets are at least twice as large). At least in this case, we can suspect a misconfiguration.

Conclusion

Being able to find out which instance of an anycast cloud you use is very important for debugging operational issues. As a side effect, it also allows funny statistics. BGP never guaranteed that you get to the "closest" instance, especially if you use "closest" in the geographical sense. The most popular instances are therefore not always the ones you expect. This assume that the labels returned by the DNS server about itself are correct, which is not always easy to achieve in the very dynamic environment of the Internet.

2 Comments

john.r.bond@gmail.com
john bond says:
Jan 22, 2014 03:18 PM
Hello Stephan,

Firstly, thank you for conducting this analysis. i preformed further investigation on the kbp01.lroot-servers.net. It appears that this node it is in fact very well connected. When I look at the probes that get an NSID of
kbp01.lroot-servers.net is see the majority of them have a directed peering with AS 20144 (traceroutes[1]) who is the host of this node. As the network is only one AS hope away it is often preferred. An example of where we see this preference is with AS 33540 where all 27 probes hit kbp01.lroot-servers.net because of direct peering with AS20144 (traceroutes[2], NSIDs[3]).

In relation to AS12322 I have been unable to recreate this issue its possible that it no longer exists. I conducted a measurement[4] to be preformed by all AS12322 164 probes. I was allocated 63 and all of them return an NSID of lys01.l.root-servers.org. Including probe 10645[5] which I believe is your probe and therefore has the same path[6] as above. As a final test I preformed the following query.

aiw6Bu6g.johnbond.net. IN A

Using tcpdump on the lys01 node i was able to see this request come in so i think we can say with confidence that this probe is hitting the lys01 node.

Another aspect which influences path choice is AS prepending. Some of our hosts prepend there AS when announcing the l-root prefix. Two examples of this can be seen via RIPEstat looking glass[7]. AS271 is prepending itself an additional 3 times at RRC03 (AMS-IX) and RRC11 (NYIIX). We also see AS35313 prepending an additional path at RRC11. These are only the examples we can see via ripe stat however we do see other examples where this occurs. L-Root hosts are free to choose how they announce our prefix so your conclusion is correct Just because an l-root node is geographically close to you it dose not necessarily mean that is the node you will use.
 
As a side note, I also audited all of the l-root boxes to ensure that there NSID is configured to the correct value and was able to find no issues. To analyse the atlas measurements i used Steaphans script[8] mentioned above for the NSID's and Jan Hugo Prins json2traceroute script[9] for the traceroute's

Thanks
John
ICANN DNS Operations

[1]https://atlas.ripe.net/api/v1/measurement/1408582/result/
[2]https://atlas.ripe.net/api/v1/measurement/1407161/result/
[3]https://atlas.ripe.net/api/v1/measurement/1406948/result/
[4]https://atlas.ripe.net/api/v1/measurement/1421320/result/
[5]https://atlas.ripe.net/api/v1/measurement/1421326/result/
[6]https://atlas.ripe.net/api/v1/measurement/1421325/result/
[7]https://stat.ripe.net/[…]/looking-glass#w.resource=199.7.83.0%2F24
[8]https://github.com/[…]/measure-dns-nsid.py
[9]https://github.com/[…]/json2traceroute.py
robert@ripe.net
Robert Kisteleki says:
Jan 28, 2014 09:02 AM
I'd also like to draw your attention to https://atlas.ripe.net/contrib/root_anycast.html?msm_id=8
Add comment

You can add a comment by filling out the form below. Only plain text is possible. Web and email addresses will be transformed into clickable links. Comments are moderated so they won't appear immediately.

Related Items
Increased Reach of RIPE Atlas Anchors

Increasing the reach of RIPE Atlas anchors is one of the highest priority goals of RIPE Atlas Team. ...

Proposing Making RIPE Atlas Data More Public

RIPE Atlas is now three years old, and is moving from a prototype to production service. Based on ...

Modifications to the IP Analyser to Reflect New Policy

We are in the process of implementing the policy regarding Post Depletion Adjustment of Procedures ...

RIPE Atlas: Improved Probe Pages

We've made it much easier to get an overview of the history and measurements for all the public ...

RIPE Atlas Fun: Map a RIPE Atlas Anchor

View maps based on RIPE Atlas traceroute measurements. Compare the maps to the ISP's description of ...

more ...