These were the three sessions on DNS over HTTPS held at FOSDEM 2019:

1. Daniel Stenberg presented a keynote session ‘DNS over HTTPS — the good, the bad and the ugly’ (video)
2. Vittorio Bertola discussed ‘The DoH Dilemma’ and
3. Daniel, Stéphane Bortzmeyer and I formed a DNS Privacy Panel, expertly moderated by Jan-Piet Mens.

I want to thank Daniel, Jan-Piet, Rudolf van der Berg, Stéphane and Vittorio for proofreading and improving this post, but I should add that this does not imply an endorsement from anyone!

Daniel Stenberg’s interpretation of what worries people about DoH

Words and definitions

During the FOSDEM presentations, various visions on the desirability of DoH were discussed. We were, sadly, rather hampered by messy definitions. In short, DoH is a transport protocol so you can securely ask DNS queries over HTTPS.

Simultaneously, Google, Firefox and Cloudflare are working on using DoH to move DNS queries from the network service provider straight onto the cloud. In other words, where previously your service provider could see (and answer) your DNS queries, in this proposed future you would send your DNS requests to a ‘free-as-in-beer’ cloud provider.

As Daniel pointed out well during his keynote, both of these things have been called DoH, which is highly confusing. ‘The Resistance’, as Daniel labels it, complains about ‘DoH’ when in fact they are mostly complaining about centralizing the DNS on cloud providers. We should not blame the protocol for what operators might do with it.

It may be that the greatest benefit we get out of the hours of FOSDEM DoH presentations is that we now know we should separate our concerns with DoH (the protocol) from our concerns about the application of this protocol to deliver what I propose, from now on, we call DNS over Cloud (DoC).

DNS over HTTPS (the protocol, DoH)

The DoH protocol is designed to use the HTTP and TLS infrastructure to deliver encrypted and authenticated DNS answers that (crucially) are hard to block by network operators. An earlier protocol called DNS over TLS (DoT) was already available but since it runs on port 853 and ‘does not look like HTTPS’, network operators that dislike DoT can easily block it. Most corporate networks will do this by default.

DoH shares the benefits and downsides of HTTPS. It sends out more trackable data than regular DNS, simply because HTTP supports things like headers and cookies. TLS session resumption functions as another tracking mechanism. On the plus side, anything that can cache or redistribute HTTPS can now also be used to improve or proxy DNS. Also, DoH makes it possible to push DNS answers even before they are asked, which could increase page load performance.

It may be seen as good or bad that HTTPS can be made undetectable and unblockable, depending on who you are and what you worry about. If Google were to co-locate a DNS over HTTPS service on the IP address also used for ‘Google.com’, economies and network operators would face a Solomon’s choice if they wanted to block DoH: give up Google searches or keep DoH alive.

Network operators that feel they should be in control of their network will not like this standoff, while users that think their network operators should have no power over them will rejoice. For the second group, at FOSDEM we discussed the proverbial Turkish dissident that would benefit from unblockable DNS.

Finally, because DoH uses authenticated HTTPS, we know we are talking to the nameserver we want to talk to. It protects against rogue nameservers, possibly injected by hijacking the DHCP request, or simply by spoofing IP packets.

DNS over Cloud (DoC)

As it stands, network operators (ISPs, service providers, and your Wi-Fi providing coffee shop) can see your DNS traffic. In addition, they could (and actually often do) manipulate or block certain queries or responses. This is an intrinsic property of providing DNS service — everyone that provides DNS service to you can do these things, cloud-based or not.

One concrete difference between typical network DNS and DoC is that network DNS tends to be unencrypted while DoC can encrypt the transport component. And encryption is good.

Much like using a VPN to access the Internet only moves your traffic from one place to another, choosing a different DNS service does not magically make your DNS more secure. It does change who you want to trust though. And if you are lucky, your trusted provider is more secure in ways that are relevant to you.

Currently, that trust is not very intentional. Internet users often have little choice in what ISP to use. In many cases, they may not even know. While local regulations (like GDPR, NIS, ePrivacy and EU telecommunications directives) may limit what a provider is allowed to do, users may not be sure if their actual network operator adheres to these legislations.

DoC proponents advocate that the user did, however, consciously choose a browser and the browser is therefore in a good position to suggest or even pick a DNS provider for their users. Users sometimes also can’t pick a browser either, but they may have the freedom to select a phone and different brands of phones include different browsers. Cheaper phones all ship with the same browser, however.

During our DNS Privacy Panel, it was also established that we estimate that most users do not care very much about their DNS privacy, and are, in any case, not well informed about the tradeoffs. The choice of DNS provider, therefore, needs to be made for them, either by their phone, their operating system or their browser.

A brief interlude on DNS encryption

Everyone agreed more encryption is good. This can happen between client equipment and the nameserver, or between the resolving nameserver and authoritative nameservers. Warren Kumari from Google tested the waters for our thoughts on opportunistic DNS over TLS between phone/browser and DNS service, and this went down well, except that it was noted it is subject to downgrade attacks. I noted that PowerDNS has been stimulating its customers to enable DoT already because Android Pie will attempt to use your DoT service if it is there. There was also a brief discussion on the efforts to encrypt traffic between the resolver and authoritative servers, something that is also good.

What does DoH protect against - if we use HTTPS anyhow?

At the very end of Daniel’s keynote, a question was asked on what the point is of protecting DNS queries and responses. The DNS response leads to the setup of a TLS connection and this TLS connection is itself already encrypted and private. We don’t need DNS for that. In addition, a TLS connection setup will typically include the name of the site being visited in plaintext, even with TLS 1.3 (the Server Name Indication or SNI field). Finally, the IP address we eventually end up connecting to may give a very good indication who this connection is going to. So it is generally possible to tell where a TLS connection is going — even without looking at the DNS. Stéphane’s RFC 7626 discusses many of these tradeoffs.

As of February 2019, there is little privacy differential when using DoH since the name still travels in plaintext. It may, however, be more expensive for a snooping network provider to extract the SNI from packets. Also, work is ongoing to use encrypted DNS to encrypt the SNI field too, in which case DoH would actually give us more additional privacy.

What DoH does, however, deliver today is protection against DNS-based censorship.

Censorship and things that break

The PowerDNS DoC service quickly gained thousands of users, many of whom are in Indonesia. PowerDNS learned that Indonesian ISPs perform a lot of blocking and DoH servers are a great way around such blocking. It may be that doh.powerdns.org is small enough to fly under the radar of the Indonesian censors.

Separately there was a brief discussion on how DoC can break things like VPNs and split horizon. We did not explore this much further except that it was noted it actually breaks things in production. An open question is if the encryption is worth the amount of breakage observed and if we could maybe work around it.

Differences between Cloud and Network DNS Providers

The highly regulated nature of service providers, at least here in Europe, is a double-edged sword. It restricts what ISPs can do with your data but it also means they respond to court orders that block content and may implement blocking of child pornography even without such orders. Internet users may not be happy with such blocking, either because they want easy access to Torrents, or simply because they object to the very principle of communications being blocked.

In addition, while (European) service providers are under legal obligation not to monetise or otherwise sell your traffic (without very explicit permission), that does not mean they don’t do it. Specifically, all service providers here will respond to government (bulk) interception orders, and provide police and spies with a full copy of all your traffic, including the unencrypted DNS parts.

Cloud providers meanwhile are very adept at navigating the GDPR waters and are able to simultaneously promise you they won’t sell your data but also power most of their bottom line selling advertising based on what you do online. In addition, they are relatively out of reach of government interception or blocking orders, which take many months to travel to a foreign jurisdiction and frequently never arrive.

Differing views on the panel

We were lucky to have three speakers with three informed but very different opinions on the subject.

'DNS and Privacy' panelists: Bert Hubert, Stéphane Bortzmeyer and Daniel Stenberg (from left to right)

I (Bert) make my living selling software and services to telecommunication service providers. I know many European ISPs intimately and I do not believe they are engaging in secret user profiling. We have enough trouble with GDPR as it is to get any kind of DNS debugging data out of our customers. So my belief is that while service providers may not be ‘a force for good’, I do predict they’d have a very hard time breaking regulations to secretly run a surveillance economy. But, these people pay my company good money so I am biased to like them. I do not believe it is a good idea, however, to send a record of every website I visit to cloud providers like Google or Cloudflare.

Stéphane meanwhile is highly knowledgeable about how governments actually regulate the Internet. He even wrote a book on it that is subtitled ‘The Internet — a political space’. In his opinion, GDPR and other regulations may be great, but enforcement is scarce as data protection agencies do not understand DNS and do not prioritise it. This leaves room for even European service providers to sell and monetise DNS data. In addition, Stéphane is worried that when governments DO finally get interested in DNS, it is for censorship purposes.

Daniel offers a perspective inspired by his background in HTTPS — he sees the obvious benefits of not only encrypting DNS data but also authenticating its server. ‘You know who you are talking to’.

Furthermore, he observes correctly that users spend time on different networks, and that we can’t possibly expect them to study the privacy practices and reputation of every school or coffee shop where they use Wi-Fi. If users picked a suitable DoH provider that worked over all networks, they’d receive a constant level of trust — no matter what network they are on.

Daniel has separately argued that he regards an explicit promise from a cloud provider not to sell your traffic as a stronger guarantee than passively trusting that a provider will stick to the applicable laws. Finally, Daniel notes correctly that GDPR does not protect you if you are connected to a rogue nameserver (so not the one you were expecting to use). It may not be the service provider that spies on you but someone else on the path TO that provider. DoH protects against that scenario.

Who gets to pick who we should trust?

If a browser decides to use DoC for its lookups, which provider should it offer? Early in the discussion, it was noted that there should be a transparent process for deciding who could be offered as a provider, where it was also noted that this process for Firefox has been far from transparent or even operational so far. A member of the audience spotted an interesting analogy with the CA/Browser Forum that has been used to determine which certificate authorities are to be trusted. Daniel noted that this is, however, also similar to search engine selection in browsers ‘and everyone picks the default, and that is the one that pays most’.

Stéphane is of the opinion that there should be many DoC providers to choose from, but since picking one is hard, the browser should present a list with a random one at the top. This allows choice but also prevents needless concentration if a user picks the default.

Why are cloud companies so anxious to host our DNS?

Warren Kumari from Google gave a very clear response — a better and faster Internet leads to more Internet use and more searches and therefore more advertisements and thus more money for Google. Such honesty is rare and it is appreciated. Warren also reconfirmed (as happened at FOSDEM 2018) that 8.8.8.8 sticks to its privacy policy, it is not being mined. As an aside, when 8.8.8.8 was launched, the state of ISP DNS was indeed dire. The DNS in many service providers did not  have a good home — fitting awkwardly between network and application departments. The creation of 8.8.8.8 contributed to the vastly improved DNS we see today, and at the time it was necessary.

But why 1.1.1.1 or 9.9.9.9? Christian Elmerot from the Cloudflare (1.1.1.1) resolver team offered the explanation that people on 1.1.1.1 will get (slightly) faster answers from 1.1.1.1 for Cloudflare domains than when using other resolvers, and this makes their services more attractive.

It may, however, be that public DoC providers are not entirely disinterested in getting a copy of the world’s browsing behaviour.

Is it any faster?

DOH (or actually, DoC provided by Cloudflare) is actually 7 milliseconds slower on average than the system resolver, according to measurements performed at Mozilla back when Daniel still worked there. He does note, however, that the worst case performance of the Cloudflare DoC is actually much better than the worst case system resolver performance.

Should you run your own resolver?

It may be slower, but Stéphane noted that having your own resolver on your own machine is actually also not good for your privacy, since authoritative servers will now see your personal IP address instead of the service provider’s IP address. However, it does offer full control — at a possible performance and privacy penalty. Stéphane notes that a mixed mode local resolver, that uses a DoH provider for cache misses, may be an optimum.

What about the EDNS Client Subnet?

There was a brief and somewhat angry discussion between me and Daniel that somehow got cut from the end of the video recording. This discussion was about the EDNS Client Subnet, and how it impacts your privacy when used by a service provider.

Some large-scale ISPs include part of a customer’s IP address when sending queries to (for example) Akamai or Level3. This is currently necessary because these large scale CDNs perform load balancing via DNS and they need to see 24 or sometimes even 25 bits of the IPv4 address to determine the right server for a user.

This is sometimes reported as a privacy problem, and in the general case, it could be. However, when used between a hosting provider like Akamai and a service provider, there actually is no loss of privacy — the customer is attempting to connect to an Akamai service, and Akamai will always see the subscriber IP address in that case.

The balance

This is where the attempt at impartiality in this post ends.

First, we should separate a few things: DoH, DoC and ‘DoC-by-default’. It seems clear that the first two of these are not problematic. It is good that we have a secure DNS transport mechanism and some DoC providers may truly be a step up in privacy and security for users in some economies or places.

Our discussion should be about what we think of ‘DoC-by-default’, that is, any attempt by browser vendors to default people into moving their DNS to Cloudflare or themselves. My concern also extends to a weaker form where you get DoC-by-nudge if you press a little ‘Got it’ button when prompted if you want to benefit from the ‘Google Secure Lookup Service’.

Who should we believe — the highly regulated (European) service provider that says it is not allowed to spy on its users using DNS, and also says they aren’t doing it? Or should we believe the cloud provider that claims those service providers are spying on their users and then asks us for our DNS traffic while promising not to sell it, although they will log each and every query?

Is it credible for DoC providers to spend huge efforts on pushing their DNS services but also claim they won’t do anything impactful with the results? Google has advanced that faster DNS means more revenue for them, which is likely true, but DoH will first slow things down!

Cloudflare meanwhile claims that if you use their DoC-service, this makes accessing Cloudflare domains a tiny bit faster compared to visiting competitors’ domain names. While this may be true, first the effect is tiny and second, it is not that great a sell for an actual Internet user.

Meanwhile, it is currently true that your coffee shop Wi-Fi may be spying on you, or may enable an attacker to do so. However, as noted above, the names of sites you visit are still sent out unencrypted by TLS connections these days, so DoC does not even deliver on saving you from such spying right now.

Finally, much is made of the users in repressive regimes that might benefit greatly from unblockable DNS, and this may indeed be so. But as noble as it is to help the Turkish dissident to communicate with the world, it seems odd that to help her we need to send a record of every website visited by 500 million Europeans to Cloudflare!

Other arguments for DoC-by-default may be more appealing — if you are a cloud provider. Despite all promises to not sell the DNS data, not log it for very long and so forth, the fact remains that a DoC operator gets sent a copy of every server and site name a user visits. Somehow, someday, that data is going to be monetised, and this will happen in ways users will not be consulted about.

This leaves us with other explanations for the DoH push, and none of these are very good. For one, it is plain and simple an attempt to fully control the Internet experience. As an example, there have been ISPs that have pondered adding ad filtering as a network service. This does not sit well with advertising companies like Google, so they’d love to be sure such blocking is not possible. DoC-by-default gives that to them.

We’ve previously established that users do not have strong or informed opinions on the source of their DNS, so whatever happens will be decided by browser vendors, on behalf of Internet users.

Given what we now know about the relative risks and benefits of DoC, in my opinion it seems utterly unwarranted to decide that users should give their DNS to Google or Cloudflare because there is no credible claim it will actually improve their lives.

This article was originally posted on the APNIC blog and has been adapted from original post which appeared on the PowerDNS Technical Blog.