Participate in our cross-vendor survey.

Why do vendors need feedback?

The DNS protocol has been with us for 33 years now and its complexity is daunting! Its specification has grown from 132 pages in 1987 to 3,000 plus pages at present, and it keeps growing! DNS software offers lots of vendor-specific features as well, which adds even more complexity. This complexity, in turn, makes manuals longer, configuration more error-prone, and software more buggy and less reliable.

In theory, vendors could decide to remove obsolete features and code, making the appearance of bugs less likely … if they only knew which features are actually used by their users. Getting rid of obsolete code would help both parties. But it is exactly this kind of feedback from administrators which is missing. Vendors who try to be conservative keep adding options while not removing anything, and that is simply not a feasible long-term strategy.

What does this historical baggage look like in practice? We looked at the documentation for various software packages, estimated the number of options, and compared the total number of options with the usage indicated by 120 detailed survey responses received till date.

BIND:

$ man named.conf | sed -e 's/ //g' | sort -u | wc -l

BIND 9.16 named.conf supports roughly 400 plus options, and many of these can be used in various contexts (global, view, zone) and interact together, also with an authoritative DNS server which is part of BIND. At the moment the survey data shows that only 65 out of 400 options are used in practice.

The candidate for the most obscure option not yet seen in the survey is: “dialup” with six modes to chose from. Is anyone still using it? (Probably yes.)

Unbound:

$ man unbound.conf | grep '^ *[a-zA-Z0-9_-]*:' | sed -e 's/ //g' -e 's/:.*$//' | sort -u | wc -l

Unbound 1.10.0 unbound.conf supports roughly 230 plus options, but at the moment our survey shows only 30 options in active use, possibly because only few Unbound administrators voluntarily submitted their configuration in the survey.

The candidate for the most obscure option not yet seen in the survey responses is: “dlv-anchor”

PowerDNS Recursor:

$ pdns_recursor --help | fgrep -- -- | wc -l

PowerDNS Recursor 4.3.0 comes with 152 options in its configuration file, which is a significantly lower number, but it also has a built-in Lua interpreter for configuration, making its configuration file Turing-complete.

At the moment our survey does not have enough data from PowerDNS users, but it has a candidate fort the most obscure option: “distribution-pipe-buffer-size“.

Knot Resolver:

Knot Resolver 5.1.1 is the newest kid on the block, but its innocent-looking configuration file is practically a Lua program with infinite possibilities. Currently the survey data shows that some users actually do use Lua for scripting their own functions inside the resolver, but the majority of respondents use only the pre-baked functions shipped with the software. This opens the question - is a Lua configuration is worth the complexity, or can it be replaced with something more user-friendly?

The candidate for the most obscure option not yet seen in the survey responses is: modules.unload(‘detect_time_jump’)

As you can see, all four implementations have vast configuration possibilities – that’s a lot of code to maintain and test, especially because the features often interact with each other. At the same time, our survey suggests that a lot of options might not be used, which opens the possibility to remove historical cruft. Please participate in the survey, it will help to determine which obsolete parts should be removed to eliminate bugs and simplify configuration.

How bad is the lack of feedback?

To illustrate the scale of the problem, we made a back-of-the-envelope estimate to see how many operators provide feedback to DNS resolver vendors (see more details about the methodology we used in the appendix below).

We estimated that around 1,330 people talk to their DNS vendors. We further estimated that the number of DNS resolver operators range from 67,000 to 8,000,000 depending on how we count.

Finally we can estimated how many operators talked to a DNS vendor in 2019:

• Upper bound = (1,330 people talking to vendors in 2019) / (67,000 Autonomous Systems) = 2 %
• Lower bound = (1,330 people talking to vendors in 2019) / (8,000,000 source addresses) = 0.017 %

We can concluded that only very few operators talked to a DNS resolver vendor in 2019, so the vendors lack feedback and are left with the following options:

• Keep maintaining all features, including unused ones, thus producing software which has more bugs and is harder to configure for everyone.

• Remove features which are not used by the small fraction of “talking” user population, possibly removing features other users depend on.

If none of these options sound appealing to you, participate in the survey and help us fix that!

Better talk late than never

The Survey is open until 30 June 2020 and gives you an opportunity to tell vendors what features users need and should not be removed, and to express wishes about further development of configuration interfaces, DNS-over-TLS and DNS-over-HTTPS support etc.

Of course, a manual survey based on a web page has significant limitations, so the survey itself also mentions the possibility of built-in “call home” features which could automate future surveys.

What do you think? Tell us!

Appendix

Guess no. 1: Number of people talking to vendors

Here we use public sources to estimate the number of people who actually talked to their vendors.

All four projects have public mailing lists, so we can download archives and use some regexes to get the number of unique e-mail addresses:

$ grep -o -h '^From [^ ]\+ at [^ ]\+ ' *-users/2019-*.txt | sort -fu | wc -l

This gives us 534 e-mail addresses including contributors working on all four projects.

Also all four projects have public bug trackers, so we can count the number of users who reported issues or commented on them in 2019. To make this task feasible we will simplify the analysis:

• We will not attempt to subtract interactions by vendors themselves.
• BIND and PowerDNS do not separate their repositories for recursor and authoritative server, so we will count all communication in these two repositories.
• We will not de-duplicate people between GitHub and private GitLab instances used by different vendors.

A simple script based on GitHub API and Gitlab CSV export produces 400 accounts posting at least one comment on public trackers in 2019 (certainly an overestimate).

Lastly we also need to count customers talking to vendors in private, which is much harder to do. Luckily ISC publishes a detailed annual report which reveals that roughly 100 more customers could be talking to ISC in private. Other vendors do not publish these numbers so let’s extrapolate ISC numbers to all four vendors and add 400 more people talking in private.

Finally we can summarise the total number of people talking to vendors in 2019:

• Public mailing lists = 530 (including vendor employees)
• Public bug trackers = 400 (without de-duplication across projects)
• Estimated number of customers talking in private = 4 * 100 = 400 (extrapolated from ISC)

The total is 1,330 people which is almost surely an overestimate… but how does it compare with number of DNS resolver operators?

Guess no. 2: Number of operators

It's impossible to obtain a precise number, but we can establish a range of possible values.

A very conservative lower bound could be number of Autonomous Systems in use on the Internet. At the moment roughly 67,000 operators care enough about Internet infrastructure to run their own AS, and are thus likely to operate other essential Internet services like DNS resolver.

An upper bound is much harder to establish. If we limited ourselves to recursive DNS resolvers we could base our guess on number of unique IP addresses sending DNS queries to DNS root over period of one day, but number of unique source IP addresses calculated over all root server instances is not available. We need to resort to independent statistics of each root server operator. From these we can see that L-root seems to have the highest number of unique source IP addresses seen during a day, varying around 8 million.

This gives us very broad range from 67,000 to 8,000,000 DNS operators.

This article is based on a blog post originally published on the CZ.NIC blog.