The Joy of TXT

Probably the most common way that the Domain Name System (DNS) is described is as a way to map domains to IP addresses. But this only tells part of the story: domains can have all sorts of records associated with them other than just IP addresses. These different resource record types, known as RRTYPEs, are used for things like aliasing a domain to another one (CNAME and DNAME records), specifying which mail servers to use for a domain (MX records), or securing domains against poisoning attacks with DNSSEC (DS, DNSKEY, et al.) - and a whole bunch of other things.

One of the most common, and versatile, types are TXT records. They can contain any arbitrary text, and domains can have multiple TXT records associated with them. Each TXT record can also have multiple strings associated with it (see below). They’re often used to configure mail security for a domain, verify domain ownership, and just to record various bits of information.

Because they’re arbitrary though, they can really be used for anything. So I decided to take a closer look, by checking the TXT records for 1 million domains and seeing what I could find.

TXTual history

NB: If you’re not familiar with RFCs, they’re kind of like standards for the Internet - they codify common practices. See Wikipedia for a full description of what they are and how they work.

Due to their flexibility, it's hard to predict how TXT records will be used. But where did they come from? In 1987, TXT records were first defined in RFC 1035 as "descriptive text". The only note provided was that "the semantics of the text depends on the domain where it is found". So that left it pretty open.

Later, in 1993, RFC 1464 was published - "Using the Domain Name System To Store Arbitrary String Attributes". This formalised the use of TXT records to store configuration settings for a domain in the format "key=value". While this format wasn’t required when using TXT records in this way, it definitely seems to have become the most common method used since then.

Another common use of TXT records is in other RFCs. Because of their flexibility, TXT records can be used as a place to store details about a protocol or framework, as in RFC 7208 (Sender Policy Framework). These types of RFCs specify the exact values and format that should be used, as an alternative to defining a whole new RRTYPE like with LOC.

Notes on the format of TXT records

Your basic TXT record looks like you’d expect - a simple character string:

Domains can also have multiple TXT records associated:

There is no way to specify any kind of ordering, so multiple records can be returned in a different order each time you ask for them.

But as mentioned earlier, it’s also valid to specify multiple strings:

Something I discovered while writing this is that RFC 7208 (Sender Policy Framework) has an interesting definition of how this usage is interpreted:

I think this applies to all DNS records, but it might just be for SPF in particular. If I’m right, then this doesn’t appear to be a well-known fact, because a lot of domains out there specify multiple strings but seem to assume that a space would be added between them when concatenating them together. One particular example is for one of the DNS giants of the Internet, Akamai, who have the following set for akamai.net:

Which, according to RFC 7208, should end up as:

Move along.

TXT work (source data and methodology)

I wrote a pretty basic shell script that I’m not particularly proud of, but worked well enough, to go through two different lists of top domains and check TXT records for each entry:

The general intent was to ensure the output was human-readable, because I wanted to be able to look through it myself, but also useful to allow for parsing to get totals etc.

I then ran this and captured the output:

This produces output in the file host.out for each domain that looks like this:

The script took about 48 hours to run each time against a nameserver that hadn’t been specifically warmed on the results. The first run was with the Tranco list, generated on 10 March 2023, available at https://tranco-list.eu/list/PZJ3J. This wasn’t as interesting as I’d hoped though, I think because it was all effective second-level domains (eSLDs, which I call “parent domains”). So I ran it again with the Cisco top 1m domains list downloaded on March 13th 2023.

I did a lot of checking with just cut, grep, etc., but I did also then write a little script to import all records into an SQLite database to make things easier.

All of the files mentioned here, including scripts, the SQLite database, and files created while looking at record lengths, unique records, etc., have been uploaded to GitHub under a Creative Commons Zero licence:

• https://github.com/pgl/joy-of-txt

TXT by numbers

Conclusion

So, what are TXT records used for exactly? Well, we can see that key-value settings are the most common use case, with domain verification records being the majority of those. SPF records also make a strong showing, as well as a lot of seemingly random fixed-length records that are probably being used for encoding data somehow.

But overall, they really are used for anything and everything. There’s some patterns we can pick out, but the lack of rigid rules means that the freedom to put whatever you like in a TXT record has been liberally accepted by the Internet as a whole. Which, if you ask me, is a good thing - having something in the DNS that can act as a config store, notes field, playground for new standard, or even the basis for file storage (not that this would really be recommended), has meant that we haven’t had to wait for standards to catch up in order to continue making use of this wonderful system that underpins a fundamental part of the Internet.