I was at FOSDEM 2016 last weekend, and wrote up a few of my observations around DNS there. Enjoy!
What is FOSDEM?
FOSDEM
is the Free and Open-Source Developers European Meeting. Every year the Université Libre de
If you are an open source developer, I recommend attending FOSDEM. It’s awesome.
DNS at FOSDEM
While DNS has a lot of proprietary implementations - both products and services - there is a lot of DNS work that is open source. Since I am a DNS person, I spent some time at various DNS talks. These were spread across a lot of different sessions; there was no DNS or even tracks or developer rooms. I surely missed some talks though, given the scale of FOSDEM, but hopefully I hit most of the interesting ones.
systemd
The opening plenary talk was by Lennart Pöttering, well-known for his previous work on PulseAudio and more recently for his work on systemd .
systemd has been adopted by most major Linux distributions (Debian, Ubuntu, Fedora, CentOS, …). It has been the source of controversy, partially because it replaces a lot of system functionality that has historically been provided by other means, such as the venerable System V init script system.
The title of the talk was quite generic, and I was pleasantly surprised to learn that about 80% of talk was actually about DNS and DNSSEC.
systemd is known for re-implementing existing functionality, and it turns out that DNS resolution is no exception. The developers are busily implementing systemd-resolved. This may be a good idea, or not. DNS developers that I talked to seem to regard this as a bad idea, although the reasons for this vary. (Personally I think that re-implementing the protocol is a classic beginner’s mistake with DNS. It looks like a simple protocol, but very quickly the complications of real-world, modern DNS become daunting.)
It turns out that Lennart is a convert to DNSSEC, and even though he reports that only 2% of DNS is secured with DNSSEC systemd is going to validate responses.
Lennart did give a good overview of some of the problems with implementing DNSSEC on the resolver of a user computer. These include:
1. Private DNS zones (for example FRITZ.BOX) don’t work
These get a authoritative NXDOMAIN reply, and so can’t be seen. The systemd solution is to special-case NXDOMAIN for TLD and allow that.
2. Broken resolvers
The systemd resolver uses the local resolver provided by the network, and often times these fail in weird and wonderful ways when a client tries to use DNSSEC. systemd will turn of DNSSEC in these cases.
3. Captive portals require registration
If you are a hotel is redirects browser lookups to a registration page. Part of this often involves returning bogus answers to DNS queries. systemd works with Network Manager or systemd networkd or whatever other tool is used to allow DNSSEC to be disabled until registration is complete.
The basic approach is “downgrade aggressively by default”. Users can require that DNSSEC always be applied, but in the current Internet that seems to be too difficult by default. :(
I did ask about TLS or other privacy plans, but Lennart responded that DNS is public. I pushed back on this given pervasive monitoring, and Lennart said that he did not care because servers do not provide TLS and he only wants things that he can use. This seems like a fair, although disappointing and short-sighted, approach.
Knot DNS Resolver
Ondřej Surý used a lightning talk to present CZ.NIC’s latest work in the DNS software space: the Knot DNS Resolver .
It’s GPLv3+, which is nice. (The only other GPL DNS resolver is the PowerDNS Recursor. Personally I am a fan of copyleft licenses, since they limit the ability of companies to incorporate it into proprietary products, giving little or nothing back to the community.)
The Knot DNS Resolver is written in C and LuaJIT. An audience member asked why it is written in C, and the answer was “because we like C”, although Ondřej noted that the Knot server is written in C. Ondřej also mentioned that CZ.NIC takes care to insure that it is written and tested in as safe a way as possible.
The software uses no internal threads, relying on multiple processes for scaling. There is a long list of supported features, some not implemented anywhere else, such as QNAME minimization.
Especially unique about the software is the cache design. It supports flexible cache backends. By default this is lmdb, but memcachedb and redis are also supported. One benefit of this is support for a persistent cache.
Anyone interested in seeing a pretty web view will be happy to know that it supports a tinyweb module. You can see a demo of this here .
Managing a complex DNS environment
One of the Facebook engineers, Stephan Gorget, gave a presentation to a standing-room only crowd talking about how they manage their DNS.
He gave an overview of their service, and then started with an inspiring quote:
Managing DNS is easy.
- said no one ever
The rest of the talk was split into two parts, one covering the internal DNS and the other the external DNS.
Internal DNS at Facebook
Facebook internal DNS runs Unbound, which is not surprising, and also TinyDNS, which is less expected. Apparently the TinyDNS zone file format is a compelling feature in the choice of TinyDNS versus BIND 9, which was the only other DNS server mentioned. For a DNS professional it certainly seems like an odd choice, and may be part of Facebooks resistance to adopting DNSSEC (this is just my theory - nobody asked about it).
On the internal side there are both human-generated and machine-generated changes, and a pipeline to reconsile changes and create a final version. A torrent seed is used to push zone changes out to all internal servers, which is a nice touch.
External DNS at Facebook
On the external side the main issue is getting users to the best site to deliver them data as quickly as possible.
Facebook has a clever system which manages TCP and TLS termination near users, and then tunnels requests for data after that to more distant PoP if necessary. This setup can save several slow packet round trips. The example gave was turning a 600 msec time into 240 msec - still not very fast but undeniably a massive savings.
Facebook uses a trick where some small percentage of users are given a unique name to look up, which is then matched on the HTTP side. This allows Facebook to build a map of the relationship between users, resolvers, and their infrastructure. This map is updated every 2 minutes, and send to their DNS servers at that time.
Talk message points
Facebook’s engineers seem justifibly proud of their creation. It is a load-balancing DNS service that automatically generates and updates configration, with direct user benefit.
They give the following recipe for success, which seem applicable to almost any DNS setup:
- Use simple sevices as building blocks
- Keep no state in your system - or as little state as possible
- Focus on reliability, reproducability, and traceability
The promise they make is that this will make things “easier and more complex”. :)
DNS ad-hoc gathering
In addition to presentations, Peter van Dijk organized an ad-hoc gathering for anyone interested in DNS. It’s always nice to chat with people working on DNS about the problems they are seeing as well as things they are excited about.
There ended up being about 10 people who showed up. There was no fixed agenda, but topics ranged from dnsdist to LDAP, from ICANN RSSAC Caucus to Paul Hoffman’s DNS testing, and the always-popular observations that DNS software never gets upgraded.
If I had been more diligent I would have taken some notes, but I admit I was enjoying the conversation too much….
Other Stuff
Even with all of this DNS, there was plenty of time at FOSDEM to learn about all kinds of great stuff, including tons of IoT presentations and demos, an update about the RIPE NCC’s efforts to open the source of RIPE Atlas , CZ.CNIC’s fabby Turris Omnia router project, and an interesting talk by the insightful-but-abrasive Richard Stallman of the Gnu project & FSF.
Comments 1
Comments are disabled on articles published more than a year ago. If you'd like to inform us of any issues, please reach out to us via the contact form here.
Mihnea-Costin Grigore •
This is a great write-up Shane, thank you for sharing! The systemd discussion and the Facebook infrastructure details were quite interesting to follow...