The Resolvers We Use

Within this list of 10 visible resolvers there are only three distinct /24 subnets being used. It’s a reasonable assumption that the resolvers that are addressed from a common subnet are likely to be part of a common resolver server farm. What if we look at a slightly different definition of a visible resolver that unites all resolvers that sit on a common /24 subnet, and call the result a “visible resolver system”. What are the 10 largest visible resolver systems in that case?

In this picture of resolver systems, 6 of these 10 largest DNS resolvers are operated by Google as part of its Public DNS system. What if we group together all these Google Public DNS systems into a single resolver system? At the same time, let’s also group together the resolvers in Comcast (AS7922), OpenDNS (AS36692) and ChinaNet (AS4134), and categorise each of these as a single visible resolver system as well.

ChinaNet (AS4134) uses a different DNS server deployment framework than many other large ISPs. Instead of funneling all DNS queries into one or two very large forwarding resolver farms that then pass the queries to the authoritative name servers, the Chinanet DNS resolution architecture appears to use a highly distributed DNS resolver deployment without concentrating the forwarded queries into a smaller set of caching recursive resolvers. In this experiment some 5,000 different DNS resolvers in AS4134 were seen to query the experiment’s authoritative name server, and 1,031 of these were observed to be acting for two or more different end clients. For the ChinaNet numbers these 1,031 resolvers have been grouped into a single report line, but it is not a true like-for-like comparison with the other resolvers listed here, as these resolvers within ChinaNet appear to be acting autonomously, and are not part of a larger common cache DNS resolver system. Although earlier observations appear to indicate that these resolvers operate with a common name filter set.

Those first two lines of this table are significant. ChinaNet appears to operate with a 50% market share within China, which is equivalent to serving the DNS queries of some 270 million users, or 11% of the entire Internet user population. The overall majority of the DNS queries from this group of users are passed through ChinaNet’s diverse set of DNS resolvers.

Google’s Public DNS system is different, in that this is not a DNS resolver that is sitting within the service architecture of a particular ISP. It's a distributed service that anyone can opt-in and use. And many Internet users have done precisely that. In terms of the population of Internet users who use Google’s Public DNS, it’s certainly is a very significant actor in today’s DNS resolution environment. Across all of the anycast instances of the Google Public DNS service, some 10.5% of all users, or an estimated 270 million users, have their DNS queries passed through Google’s DNS service for name resolution.

These 10 largest DNS visible resolver systems account for the DNS queries of almost 30% of the Internet’s user population. What about the remaining 70% of the Internet’s users? What’s the overall distribution of resolver use? Figure 1 shows the cumulative distribution of users to these visible resolvers (you can click on the image to enlarge it).

Figure 1: Cumulative Distribution of Users to Resolvers

Figure 1 indicates that the 1,000 largest visible resolver systems undertake queries for 83% of users. The largest 1,889 visible resolver systems, or 0.7% of the total count of resolver systems, handle the query load for 90% of the end users. Obviously, the DNS resolution “market” is a highly concentrated environment, where a relatively small number of resolvers handle the query load for a very significant proportion of users.

This observation of DNS resolver concentration has a number of implications. If most of the queries that are passed towards authoritative name servers come from intensively used resolvers, then the local caches in these resolvers would normally absorb the majority of queries from users. Authoritative name servers therefore would see a somewhat skewed set of queries that predominately come from the trailing part of the resolver set that serve very small user populations. The larger resolvers will cache the authoritative name server’s response and serve a far larger population of users from the cached response. In other words, most authoritative name servers do not “see” most users’ DNS behaviour, as this is occluded by these large resolvers.

It also implies that any orchestrated effort to introduce changes in this small set of DNS resolvers will affect a significant proportion of the end user population. This has been evident in the DNS with the introduction of support for introduced extensions to DNS, EDNS0, and with the introduction of validation of DNSSEC-signed responses, where some 12% of users exclusively use DNS resolvers that perform DNSSEC validation.

With the exceptions of Google’s Public DNS and OpenDNS, the remaining 8 DNS resolver systems listed in Table 3 appear to be ISP-provided resolvers that service the ISP’s user base, and the population of clients served by these resolvers appear to largely match the estimates of the size of the client base served by the ISP. That observation would lead one to assume that for most Internet users the DNS resolver they use is provided by their ISP. In other words, it looks like it would be a reasonable assumption that the resolver used to pass a query on to an authoritative name service is located “close” to the end user who initiated the query, at least at the level of granularity of the ISP itself.

To what extent is this assumption of locality of users and resolvers actually the case? How “local” are the resolvers we use?

Non-Local Resolution

While 8 of the 10 largest resolver systems in the Internet appear to be ISP-based systems that service the clients of that ISP, this is not always the case when one looks at the larger picture of all end users, their users and the DNS resolvers that they use. It appears that there is a significant fraction of the level of DNS resolution that is non-local, for some definition of “non-local”.

Of the 84 million individual tests that were conducted as part of this experiment, it was observed that 34% of the time, or some 29 million tests, we observed DNS queries at the authoritative name server from resolvers that were located in a different network (by ASN) from that of the end user who initiated the query in the first place. This is a surprisingly high proportion.

But it’s useful to recall the entry in Table 3, where Google’s Public DNS service is used by around 10% of the world’s user base. If we remove this single non-local factor of use of Google’s Public DNS service from the picture then we are left with some 20 million users, or around 1 on 4 users who use DNS resolvers from a different ASN from that where they are located.

However having your DNS resolver in a different ASN is not necessarily a distant relationship. It could be that the resolver is located in an “upstream” service provider and the user and the DNS resolver are not necessarily located far from each other. What is perhaps more notable is the case where the DNS resolver is located in an entirely different country. The numbers of these cases are still significant. Some 21% of users send their queries via DNS resolvers located in a foreign country. Even when we remove Google’s Public DNS from the picture the numbers are still significant, with some 11% of users, or 9 million in the case of our experiment, using resolvers located in different countries.

Where are these users? Are there countries where the level of use of “foreign” DNS resolvers is significantly higher than others? There is certainly a very high variation, with 99% of users who were geo-located in Algeria using foreign DNS resolvers (half of this count in Algeria is due to use of Google’s Public DNS, which the other half is dispersed across other countries). At the other end of the spectrum 97.7% of users in South Korea use DNS resolvers located in South Korea. A map of the world showing the proportion of “foreign” DNS resolution is shown below.

Figure 2: Relative use of “Foreign” DNS Resolvers by Country

It’s evident that the highest levels of “foreign” DNS resolvers occur in Africa, parts of the Middle East, South East Asia and Central America. The lowest levels are found in South Korea, New Zealand, France, Greece and Finland.

However, Google’s public DNS is a big factor here. A second map (Figure 3) removes the Google Public DNS component from this foreign use map, and looks at the proportion of “foreign” DNS resolution using resolvers other than Google’s Public DNS.

Figure 3: Relative use of “Foreign” DNS Resolvers by Country

Since so many users end up using Google’s Public DNS service to query authoritative name servers, it’s worth looking at the use of this service in its own right. Figure 4 is the map of the world looking at the proportion of users in each country who pass their queries to Google’s Public DNS service.

Figure 4: Relative use of Google Public DNS Resolvers by Country

Its clear that Google’s Public DNS is widely used in central Africa, parts of the Middle East and across South East Asia. Perhaps not so obvious is 10% of Russian users, 14% of Brazilian users, 11% of Indian users, and 5% of users in both China and Great Britain also direct their DNS queries to Google’s DNS service. (The highest relative level of Google’s DNS usage was seen from the island nation of Kiribati, with 53% of the nation’ users. The largest estimated pool of users comes from China, where an estimated 25 million users direct their queries to Google’s PDNS.)

What about the reverse view? If 11% of the world’s users direct their DNS queries to a foreign resolver other than Google, then where are these DNS resolvers located?

Figure 5: Location of “Foreign” DNS resolvers

Why?

There are a myriad of possible reasons why one in every 10 users of the Internet have their DNS resolvers located in another country to the one that they are located in.

The first potential reason is the inaccuracy of public geo-location tools. This exercise has used geo-location data published by Maxmind , and augmented with the Regional Internet Registry data in those cases where Maxmind has not identified a country for the IP address. It’s great that Maxmind is willing to publish a geolocation data set, and kudos to them for doing so, but at the same time we have to recognise that geo-locating addresses is not easy, and the published database is not perfect by any stretch of the imagination. In other words there is a distinct possibility that the geolocation of mapping an IP address to a particular country may well be incorrect.

The second potential reason is the widespread use of various hybrid forms of tunneling. Relocating a device’s DNS resolver to a different country often bypasses simple geo-blocking systems for restricted content. So its not surprising to see the US be the target of a large proportion of DNS relocation as one measure to provide the appearance of a US-located client of a content service.

It's also possible that we are seeing the outcomes of various forms of malware, where the infected system has its DNS resolution queries redirected to a compromised DNS resolver that will answer a critical subset of queries with some form of synthetic response, while answering all other queries using conventional DNS resolution queries.

But why the preponderance of use of Chinese resolvers? Some 20% of all foreign resolution cases involve the use of DNS resolvers located in China. One explanation could be in the adoption of off-shore web proxies in order to circumvent some form of content firewall or filter operation. In this case the DNS would be correctly locating the user into China, but the end-user’s use of a foreign web proxy would mean that the web object retrieval would appear to originate from a different country. Other possible explanations involve the use of VPN tunneling and local DNS deep packet inspection and response spoofing, although such explanations are highly speculative.

The use of non-local DNS resolvers is so common in today’s Internet, with 1 in 3 users using DNS resolvers located in a different network than the one in which they are located, and 1 in 5 users using DNS resolvers that are located in a different country than the apparent location of the user. Given such widespread use of non-local DNS resolvers its reasonable to speculate that all of the above explanations may be at play here, and doubtless a few more as well.

Reports

Reports from this experiment are available online at http://labs.apnic.net/cgi-bin/rmap .