Detecting DNS Root Manipulation

In November 2021, Manu Bretelle (Meta) reported that users of an ISP in Mexico were unable to reach whatsapp.net due to potential route leaks of K-root in China. In his post to the DNS operations mailing list, Manu pointed out that the domain was receiving an incorrect authoritative response from the K-root server. This is highly unexpected. We believe this was due to DNS query rewriting by middleboxes.

Root name servers and middlebox interception

Root name servers form a critical point in Internet communication. They provide a list of the authoritative name servers for the top-level domains (TLDs). When a host machine on the Internet needs to resolve a domain name, it uses resolver software to perform the lookup. If the resolver has previously resolved the domain, it will maintain it in cache and respond to the request. However, if it does not have a result, it will query the root server. The root server then directs the resolver to a TLD name server. The resolver then queries the TLD for an answer, and it provides the next authoritative server to contact. The resolver keeps querying the next-in-line authoritative name servers recursively until it reaches the desired authoritative server for the domain.

Middleboxes are devices that can be used to manipulate this process. If there is a middlebox somewhere in the path between the host machine and the root server, it can intercept the query towards the root server and rewrite the response.

As such, middleboxes can be used for censorship by state actors. It's beyond the scope of this article to discuss censorship by state actors within their jurisdiction. However, that said, in the case of route leaks, queries from other countries can get intercepted by these middleboxes, especially if they are in the path to an instance of the root server.

For instance, in Nov 2021, due to a route leak from China, several hosts in Mexico were unable to reach whatsapp.net. It was determined that middleboxes intercepted the queries on route to the root instance hosted in China and replied with a bogus reply. Even though the outage was reported only for the hosts in Mexico, we observed that several probes in the US, Europe, and Africa could also reach K-root instances in China. It is unclear if they also observed intermittent outages. The outage lasted about a week. In the observable notebook below, we show the geolocation of probes over the week of the outage.

Since the last outage, we have not observed any probe hosted outside of China reaching the k-root server in China. However, we observe middleboxes rewriting queries sent to root servers in several other countries. In the rest of the article, we investigate the presence of these middleboxes using RIPE Atlas probes.

Detection methodology:

Our approach to identifying DNS responses with root manipulation relies on the underlying fact that root servers will not respond with an authoritative reply for a query. Instead, the response from root servers will consist of zone referrals to the TLD (top-level domain) name servers. In summary, we characterise a response as invalid if we receive an A or AAAA reply from the root servers. Figure 1 below shows an example of rewriting (left) and response from the root server (right). 

Experimental setup:

In order to fully understand the impact of the rewriting, we ran queries from all the probes towards root servers. We tested the interception attempts for Google and Facebook since these domains are usually censored. We also queried ripe.net, which serves as a baseline for us - i.e., if the query for ripe.net is rewritten, there is a middle-box in the path that rewrites any query sent towards root servers. We sent queries to resolve A and AAAA records using TCP and UDP.

Prevalence of root manipulation:

Our results show that only 419 probes received an A or AAAA record for the queries sent to root servers for resolution in March 2022. However, these probes are geographically distributed in 48 countries. Figure 3 shows the proportion of the affected probes relative to the total hosted probes per country. China and Iran have the highest percentage of affected probes. The results are not surprising since both countries are well-known for censorship. It is possible that, due to limited coverage, we are unable to measure the rewriting of DNS queries in some countries and networks. However, we observe that countries with a significant number of probes also have no cases. For instance, we have no cases in Japan (199 probes), Sweden (195 probes), Belgium (163 probes), Denmark (120 probes), and Greece (100 probes) for March 2022.

We further analyse the distribution of probes per ASN. We observe most of the probes hosted in Iran Telecommunication and Asiatech Data Transmission company. These companies provide Internet services to end-users, which means other subscribers also suffer from rewriting. Similarly, for the US, we observe rewriting for users in Lumen and a well-known ISP, Comcast. It is important to note here that it is not necessary that these providers are hosting middle-boxes. It is possible that the middleboxes are hosted in some other ASN which is on the path toward root servers.

Finally, to understand the impact of query rewriting on end-users, we use the ratio of affected probes by the total number of probes per ASN to estimate the number of end-users that will suffer from query rewriting per ASN. For instance, if we find evidence of query rewriting for five probes from a total of 10 probes in an ASN, we estimate that 50% of users will possibly be impacted by response rewriting. We used APNIC data on end-users per AS for the calculation. We could not map 53 (~20.7%) ASes, since they were missing in APNIC dataset. Our estimates show that the potential impact for most users is in China since we observe query rewriting for backbone networks like China Telecom and Unicom. Similarly, we find one of the major Internet providers in Indonesia (Telin) in our dataset. 

Comparison with baseline:

We further investigated if the middleboxes rewrite all the queries or only intercept responses for well-known domains (google.com and facebook.com). We found only 155 (~37%) probes in 40 countries that also respond to queries requesting A/AAAA records for ripe.net. The countries from which queries to only Facebook and Google are rewritten are Albania, Chile, Colombia, Hong Kong, Israel, Mexico, Romania, and Thailand. It is important to note that we find a lower percentage of DNS response rewriting in these countries from our initial results. For instance, in Romania, we found only three probes out of 80 probes whose queries were rewritten. Similarly, we found only two out of 37 probes in Hong kong whose queries were rewritten. It is likely that the middleboxes are not hosted in these countries and are in one of the upstream providers of the providers.

Comparison between root servers:

This section analyses the differences in response rewriting when we query different root servers. The figure below shows the queries to the k-root and i-root servers had significantly less number of query interception and response rewriting. We also find fewer cases in k-root and i-root in countries hosting probes with significant rewriting for other root servers. For instance, there were only 10 and 53 probes in Iran whose responses were rewritten when the request was sent to the k-root and i-root, respectively. There are more than 90 probes hosted in Iran whose queries were rewritten for other root servers. Similarly, for China, queries are intercepted for only 9 probes when the destination is k-root, while around 20 probes for the other root servers. The difference is negligible when we compare responses for probes hosted in Russia. Future research is required to untangle the differences between root servers.

Comparison between query type and record types:

We also find significant differences in responses getting rewritten when an A record is requested compared to AAAA records. We observe that only half of the probes (239) reveal rewriting when an AAAA record is requested, whereas almost all the probes (414) experienced rewriting when an A record is requested. Since, in our initial design, we chose local resolvers of probes to resolve the queries, it is possible that some of these discrepancies are due to local resolvers' ability to resolve different record types. Finally, we also find the difference of about half the probes when we use different transport layer protocols. Responses for 410 probes were rewritten when UDP was used as the transport layer, while only 123 probes received a rewritten reply when we sent a request over TCP. Middleboxes might have replied with a spoofed IP address of root servers for UDP. However, a response from a middlebox over TCP means an actual machine impersonating the root-server IP address is in the path towards the root server.

Conclusions

In summary, we find evidence of middleboxes rewriting the queries sent to root servers. The probes that received these replies are geographically distributed across several countries. Even though, since the last outage, we have not observed any probe hosted outside of China reaching the k-root server in China. We found four probes hosted in Mexico still receiving replies from middleboxes, one of which was reported by Manu, which suffered an outage for whatsapp.net. Moreover, probes hosted in four countries were able to reach k-root instances in Iran, where we saw a higher number of probes receiving rewritten replies. Future work is required to ascertain the downtime due to these interceptions.