A Tale of Using Public DNS Servers in Iran — Part 3

Curious case of corrupted MX response

Following my research on DNS reachability and performance (please see part 1 and part 2 of this story), I came across an interesting result with specific domain names. While looking up almost every DNS request from Google Public DNS took about 100 milliseconds, specific queries from the same DNS servers were being resolved in under 10 milliseconds. In some cases the reply was not authentic or even corrupted altogether ¹ .

Here is an example of a normal DNS query being sent to Google Public DNS (using dnsping ):

 ./dnsping.py -c 3 -s 8.8.8.8 -t A apple.com
DNSPING 8.8.8.8: hostname=apple.com rdatatype=A
100 bytes from 8.8.8.8: seq=0   time=97.000 ms
100 bytes from 8.8.8.8: seq=1   time=97.803 ms
100 bytes from 8.8.8.8: seq=2   time=90.176 ms

--- 8.8.8.8 dnsping statistics ---
3 requests transmitted, 3 responses received,   0% lost
min=90.176 ms, avg=94.993 ms, max=97.803 ms, stddev=4.191 ms

The same query for domain twitter.com takes much less time:

 % ./dnsping.py -c 3 -s 8.8.8.8 -t A twitter.com
DNSPING 8.8.8.8: hostname=twitter.com rdatatype=A
33 bytes from 8.8.8.8: seq=0   time=8.390 ms
33 bytes from 8.8.8.8: seq=1   time=12.722 ms
33 bytes from 8.8.8.8: seq=2   time=17.546 ms

--- 8.8.8.8 dnsping statistics ---
3 requests transmitted, 3 responses received,   0% lost
min=8.390 ms, avg=12.886 ms, max=17.546 ms, stddev=4.580 ms

Resolving the SOA record for the same domain yields normal results (note the ~100ms response time):

 % ./dnsping.py -c 3 -s 8.8.8.8 -t SOA twitter.com
DNSPING 8.8.8.8: hostname=twitter.com rdatatype=SOA
95 bytes from 8.8.8.8: seq=0   time=133.951 ms
95 bytes from 8.8.8.8: seq=1   time=98.027 ms
95 bytes from 8.8.8.8: seq=2   time=93.309 ms

--- 8.8.8.8 dnsping statistics ---
3 requests transmitted, 3 responses received,   0% lost
min=93.309 ms, avg=108.429 ms, max=133.951 ms, stddev=22.228 ms

Looking at the results above, a network expert would conclude that some specific queries are being handled locally.
I observed another anomaly when I tried to look up the MX record locally and noticed MX records for these domains are not resolved correctly:

 % dig -t MX twitter.com @8.8.8.8
;; Got bad packet: bad label type
45 bytes
40 10 81 80 00 01 00 01 00 00 00 00 07 74 77 69          @............twi
74 74 65 72 03 63 6f 6d 00 00 0c 00 01 c0 0c 00          tter.com........
0c 00 01 00 00 03 79 00 03 41 41 41 00                   ......y..AAA.

Repeating the same experiment using RIPE Atlas , confirmed my initial analysis and I found out most of the probes in Iran cannot resolve the MX record for the domain twitter.com and they receive a bad reply in less than 100ms.

Taking a closer look at the DNS request and response using tshark, it seems that in the answer to my MX query, I received a PTR answer with corrupted RDLENGTH and RDATA fields.

Here is the query :

 % tshark -O dns -2xnVr bad-mx-reply.pcap -R "frame.number eq 1"

Frame 1: 71 bytes on wire (568 bits), 71 bytes captured (568 bits) on interface 0 (outbound)
Ethernet II, Src: xx:xx:xx:xx:xx:xx, Dst: xx:xx:xx:xx:xx:xx
Internet Protocol Version 4, Src: 192.168.0.106, Dst: 8.8.8.8
User Datagram Protocol, Src Port: 53046 (53046), Dst Port: 53 (53)
Domain Name System (query)
    [Response In: 2]
    Transaction ID: 0x31d2
    Flags: 0x0100 Standard query
        0... .... .... .... = Response: Message is a query
        .000 0... .... .... = Opcode: Standard query (0)
        .... ..0. .... .... = Truncated: Message is not truncated
        .... ...1 .... .... = Recursion desired: Do query recursively
        .... .... .0.. .... = Z: reserved (0)
        .... .... ...0 .... = Non-authenticated data: Unacceptable
    Questions: 1
    Answer RRs: 0
    Authority RRs: 0
    Additional RRs: 0
    Queries
        twitter.com: type MX, class IN
            Name: twitter.com
            [Name Length: 11]
            [Label Count: 2]
            Type: MX (Mail eXchange) (15)
            Class: IN (0x0001)

0000  64 66 b3 87 4a e0 88 63 df 8c a7 83 08 00 45 00   df..J..c......E.
0010  00 39 bd 61 00 00 40 11 ec 30 c0 a8 00 6a 08 08   .9.a..@..0...j..
0020  08 08 cf 36 00 35 00 25 50 ae 31 d2 01 00 00 01   ...6.5.%P.1.....
0030  00 00 00 00 00 00 07 74 77 69 74 74 65 72 03 63   .......twitter.c
0040  6f 6d 00 00 0f 00 01                              om.....

And the answer:

 % tshark -O dns -2xnVr bad-mx-reply.pcap -R "frame.number eq 2"

Frame 2: 87 bytes on wire (696 bits), 87 bytes captured (696 bits) on interface 0 (inbound)
Ethernet II, Src: xx:xx:xx:xx:xx:xx, Dst: xx:xx:xx:xx:xx:xx
Internet Protocol Version 4, Src: 8.8.8.8, Dst: 192.168.0.106
User Datagram Protocol, Src Port: 53 (53), Dst Port: 53046 (53046)
Domain Name System (response)
    [Request In: 1]
    [Time: 0.007404000 seconds]
    Transaction ID: 0x31d2
    Flags: 0x8180 Standard query response, No error
        1... .... .... .... = Response: Message is a response
        .000 0... .... .... = Opcode: Standard query (0)
        .... .0.. .... .... = Authoritative: Server is not an authority for domain
        .... ..0. .... .... = Truncated: Message is not truncated
        .... ...1 .... .... = Recursion desired: Do query recursively
        .... .... 1... .... = Recursion available: Server can do recursive queries
        .... .... .0.. .... = Z: reserved (0)
        .... .... ..0. .... = Answer authenticated: Answer/authority portion was not authenticated by the server
        .... .... ...0 .... = Non-authenticated data: Unacceptable
        .... .... .... 0000 = Reply code: No error (0)
    Questions: 1
    Answer RRs: 1
    Authority RRs: 0
    Additional RRs: 0
    Queries
        twitter.com: type PTR, class IN
            Name: twitter.com
            [Name Length: 11]
            [Label Count: 2]
            Type: PTR (domain name PoinTeR) (12)
            Class: IN (0x0001)
    Answers
        twitter.com: type PTR, class IN
            Name: twitter.com
            Type: PTR (domain name PoinTeR) (12)
            Class: IN (0x0001)
            Time to live: 889
            Data length: 3
[Malformed Packet: DNS]

0000  88 63 df 8c a7 83 64 66 b3 87 4a e0 08 00 45 00   .c....df..J...E.
0010  00 49 00 00 00 00 3b 11 ae 82 08 08 08 08 c0 a8   .I....;.........
0020  00 6a 00 35 cf 36 00 35 fb c6 31 d2 81 80 00 01   .j.5.6.5..1.....
0030  00 01 00 00 00 00 07 74 77 69 74 74 65 72 03 63   .......twitter.c
0040  6f 6d 00 00 0c 00 01 c0 0c 00 0c 00 01 00 00 03   om..............
0050  79 00 03 41 41 41 00                              y..AAA.

You will notice the anomaly in the reply shown above. While the query was made clearly for an MX (Type 15) record, the replied packet contains an answer for PTR (Type 12) record. Also the “Data length” field is 3, while the payload contains 4 bytes (in the last line in the packet dump you can see “03” followed by “41 41 41 00”).

Trying the same query from any other host around the world, I could not reproduce the same result. Therefore I assume this is only happening inside Iran.

Later I used the list of top 10,000 domains (according to OpenDNS) to figure out if this anomaly is happening for any other major domain. So I wrote a quick script to look up MX records for all the domains and foundnd corrupted results.

The tests confirmed that the same anomaly was seen on 139 out of 10,000 major domains ² .

Conclusion

It would be a good idea to have local instances of major DNS resolvers to improve user experience. However, a misconfigured DNS server or any network device may have huge impacts on end user experience.
The anomaly we discussed in this report could have enormous negative impact on email traffic and may avoid delivering emails to certain domains.