Measuring the Impact of DNS Flag Day

In the lead up to DNS Flag Day, and as part of the outreach, the focus for measurements has been the authoritative nameservers that needed to be fixed. In this post, we will take the other perspective and look at resolvers and resolver implementations — what was resolver behaviour on the Internet before DNS Flag Day, and how does the uptake of dropping workarounds disseminate in the wild?

First Flag Day — 1 February 2019

The first Flag Day was held on 1 February 2019. The focus of this inaugural event was to address the dropping of EDNS requests by authoritative nameservers.

EDNS is an extension mechanism to the DNS introduced in 1999 (RFC 2671). It allows DNS messages to:

• Be larger
• Have more return codes with DNS responses
• Extend the DNS header with more bits
• Extend the DNS with not yet foreseen future features and improvements

Many such extensions have been defined since 1999, including DNSSEC, which also depends on EDNS for its operation.

Authoritative nameservers not supporting EDNS

Authoritative nameservers that do not understand EDNS are expected to send a response with RCODE FORMERR (RFC 2671 updated by RFC 6891). In this way, nameservers that aren’t known to support EDNS can be retried without EDNS immediately.

Figure 1: The issue: authoritative nameservers not indicating they do not support EDNS

Unfortunately, in practice, some authoritative nameservers without EDNS support drop the request and do not reply at all. This is disastrous for the resolver because such a situation is indistinguishable from packet loss or an authoritative nameserver that is down. The only way for a resolver to interact with such an authoritative nameserver is to retry after a certain timeout without EDNS. This makes resolution unnecessarily slow and inefficient.

All newly released versions of BIND, Knot, PowerDNS and Unbound no longer have these workarounds in place and will consider a non-responding authoritative as down — in this post, we will call such resolver behaviour as strict versus permissive, which have workarounds in place. The public DNS operators would immediately or incrementally move from permissive to strict behaviour.

To be able to measure the uptake of the changes that were announced to happen after Flag Day, we at NLnet Labs, SIDN and Rochester Institute of Technology created an on purpose, non-compliant authoritative nameserver that drops any EDNS query. This nameserver serves the flagday.rootcanary.net zone.

Figure 2: On purpose non-compliant authoritative nameserver

Resolvers that are able to look up names (such as test.flagday.rootcanary.net) within this zone are permissive.Resolvers that cannot are strict.

Resolver implementations before Flag Day

As a first step, we examined existing open source resolver implementations to see how they support EDNS. We found that two implementations were already NOT able to resolve names from our broken authoritative nameserver.

1. The Knot resolver never had any workarounds; it has always been strict.

2. The most recent version of the PowerDNS Recursor, available before Flag Day (4.1.10), could not resolve the test name either. Although the announcement of the Flag Day release 4.2.0-alpha1 mentions the removal of workarounds, those did not help with our test setup.

On 1 February, the latest releases of both BIND and Unbound were permissive. BIND started to resolve strict from 27 February 2019 with release 9.13.7 and Unbound from 5 February with release 1.9.0.

The effects of Flag Day as seen from end entities

On 30 January 2019, we started a measurement that uses all RIPE Atlas probes to send a query every four hours using the DNS Resolvers configured on the probes (which are by default learned from DHCP). Care was taken so that each resolver queried a unique name and would not reuse an earlier cached result from a middlebox. Thus, if we can get a valid answer from a tested resolver, the resolver would be identified as permissive, and invalid answers or no answer as strict.

As of August, we’ve managed to target 10,127 probes, which each had one or more resolvers, providing a total of 19,272 resolvers.

Remarkably, 15% of these resolvers were already strict before Flag Day. This number had risen to 42% as of 1 June 2019 (Figure 3). Since then the number has risen another 1.9% to 43.9%.

Figure 3: Uptake of strict resolving on RIPE Atlas

When looking at the ASes doing strict resolving, the Google Public DNS has had the largest impact on the rise of these numbers (Figure 4).

Figure 4: Strict resolving on RIPE Atlas uptake by ASN

Below, we have the results from the participating public resolvers, respectively, from left to right, top to bottom: Cloudflare, Google Public DNS, Quad9 and OpenDNS:

Note, these graphs show absolute numbers of resolvers. For example, the number of RIPE Atlas probes using the Cloudflare public resolver is rising, but the relative number of resolvers that are strict is not at the same rate.

Given that Cloudflare uses the Knot resolver, we expected to see 100% of strict resolver behaviour the entire measurement period, but instead, we see a small percentage of permissive resolver behaviour throughout the measurement period.

With the Google Public DNS, this is completely different. They started with a small percentage of their infrastructure doing strict resolving right from the start, and incrementally increased it from mid-March until they reached their top (almost 100%) around the third week of April.

Quad9 had +-85% strict resolving already. Nothing much happened until 1 May, when that number suddenly increased to almost 100%

OpenDNS had and still has 100% of permissive resolving. As such there was no Flag Day-related activity with them at all.

Effects of Flag Day as seen at the authoritative nameservers

Every query sent with RIPE Atlas had a unique name. At the authoritative nameservers, we can now see which packets belong to the same resolver. This enables us to identify strict and permissive resolvers from traffic seen at the authoritative nameservers. We used the algorithm expressed in pseudocode below to make the distinction:

for each unique QNAME:
    if all queries with EDNS:
        All IPs seen are strict
    elif at least one query without EDNS:
        All IPs seen are permissive

Because we don’t need to see the responses, we can trigger a much larger number of different resolvers from end-points on the Internet. We also used Open Resolvers to schedule uniquely named queries and we triggered Luminati (a VPN provider with which gratis subscribers provide an exit node from their home that we use to send queries from). Table 1 summarises the number of distinct resolver IP addresses and the number of distinct ASes seen from all of them:

Table 1: Total number of observed IP addresses at our authoritative name server

Figure 9 shows the percentage of these distinct IPs that became strict. Like with the end-entities perspective, we also saw quite a high percentage of strict resolvers before 1 February: more than 30%. We saw a slow increase of strict resolvers from +-35% on 1 February till +-40% on 1 August.

The much larger percentage of strict resolvers at the start with the authoritative nameserver measurements compared to the end-entities measurement, suggests that the popular resolvers were more permissive before 1 February. However, the end-entities perspective showed a much steeper increase in strict resolvers than seen with unique resolvers IPs at the authoritative nameserver. So the more popular resolvers are more responsible for the uptake of strict resolving than the less popular resolvers.

Figure 9: Share of strict resolvers since the Flag Day

Figure 10 shows the percentage of the top 10 most seen ASes at the authoritative server. The most clearly visible uptake of strict resolving is again done by the Google Public DNS.

Figure 10: Share of strict resolvers in the most seen ASes

The Internet got a little better

The DNS Flag Day ’19 stimulated a lot of awareness resulting in several people investigating and fixing their authoritative server. As a result, the Internet got a little better.

However, if your domain was served by authoritative servers that were broken in this way, you were already in pretty bad shape. Even before Flag Day ’19, two of the participating open source implementations could not resolve our test domain, and neither could two of the participating public DNS services.

More than 15% of the resolvers seen on RIPE Atlas probes and more than 30% of unique resolvers seen at our authoritative server were also not able to resolve such a domain before Flag Day ’19. The fact that nobody noticed might have something to do with the resilient provisioning of domains (maybe not all nameservers serving a domain were broken) and the resilient provisioning of resolvers at end-entities (you may have a permissive resolver configured next to a strict one).

Strict resolver behaviour is growing slowly though. It had the initial quick start with the Google Public DNS introducing strict resolving, but we do see a continuing slow increase both at the end-entities and at the authoritative server. Perhaps the slow increase at the end-entities is mostly related to the slow increase of probes using the Cloudflare resolver? That said, the slow increase seen at the authoritative server shows that it is not just that, but also the long tail of less popular resolvers.

Is there a need to rush to fix the things that DNS implementers and DNS resolver operators will no longer support before a Flag Day?

The slow uptake of its implementation (21.9% at the end-entities — Google being responsible for 20% — and just 4% seen at the authoritative server) plus the fact that a large portion of resolvers were already strict even before Flag Day ’19, might suggest it is not that pressing. But there is measurable uptake and there will be some point where a significant group of people will be hampered by an unfixed domain to remain undamaged.

Combining measurements on authoritative nameservers in use by provisioned domains with resolver measurements like we have done in this study could help pinpoint this more precisely. Giving a realistic projection would definitely help with reducing the uncertainty surrounding Flag Day.

Currently, the next DNS Flag Day (2020) is being discussed. No date is set as yet, but the focus will be on operational and security problems in the DNS caused by IP packet fragmentation. Check out the official website for more information on past and future DNS Flag Days.