Measuring World IPv6 Day - Some Glitches And Lessons Learned

The RIPE NCC measured World IPv6 Day in various ways as described in RIPE NCC Measurements for World IPv6 Day. One of the ways the RIPE NCC measured World IPv6 Day was to conduct active measurements from 40 vantage points to a selection of 53 World IPv6 Day participants. These measurements consisted of DNS,ICMP, HTTP and traceroute measurements both on IPv4 and IPv6. All measurement results are available on http://v6day.ripe.net . In this article we will look at some of the events we found in more detail.

HTTP 404 - Level3

When visiting the Level3 Communications website over an IPv6 connection, users didn't see the normal homepage. Instead they saw 404 document-not found error-pages over IPv6 (screenshot below), while connections over IPv4 got normal HTTP responses. This issue appeared in our data from 22:05 UTC on 7 June to 15:14 on 8 June. This was later acknowledged to be a webserver configuration mistake , that was unrelated to IPv6. While this issue was solved eventually, we were surprised to see it took several hours to be fixed.

Figure 1: HTTP 404 Error as appeared on Level3 website when accessed over IPv6

A lesson that can be learned from this:

• as with any technical infrastructure, it is essential to have good monitoring, and channels for external parties to report problems.

Partial reachability from vantage points

Some of our vantage points also had a problem reaching Level3 over IPv6 at all. This is due to some networks being partitioned, ie. if you are single-homed in one of these networks in IPv6 you can't reach the other. As can be seen in this Wikipedia article , the most notable cases of this seem to involve Cogent, but it also affects Hurricane Electric and, in this case, Level 3. Partitioning decreases the value of having one of these networks as your sole transit provider. It would be good for the networks involved to minimise this. Unfortunately partitioning is not new to IPv6, and this happens in IPv4 as well ( over , and over , and over , and over again).

Three of our vantage points had problems reaching a majority of the participants we monitored over IPv6. For two of the CAIDA Ark vantage points, one in Manila, the other one in Dakar, these problems persisted. As these are both in educational institutions, the suspicion is that in IPv6 they are only able to reach other educational institutions through peering of NRENs (National Research and Education Networks). For TTM box tt120 there was a problem with the local network configuration that quickly got fixed once it got to the attention of the people operating the infrastructure.

A lesson that can be learned from this:

• Be on the lookout for partial reachability. For instance, we could have done a better job in testing our vantage points for this, prior to World IPv6 Day.

Authoritative DNS issues - Juniper

For www.juniper.com an issue regarding DNS resolution was reported , and diagnosed as a problem with DNS glue records . Our measurement data shows this issue lasted from approximately 9:30 to 11:15 UTC on 8 June, and wasn't limited to missing glue records. As can be seen in Figure 2, both A and AAAA queries for www.juniper.com in this period resulted in a variety of responses. In about 60% of cases we saw a SERVFAIL returned to our vantage point both for A and AAAA. In about 30% of cases no DNS error code was returned, but also no resource record. The type of response was strongly correlated to the vantage point. About 10% of queries got responses with a resource record in it, possibly due to caching (we didn't record TTLs of returned resource records).

 Figure 2: Rate of DNS responses for A (IPv4) and AAAA (IPv6) for queries to www.juniper.com.

A lesson that can be learned from this:

• Things going wrong on World IPv6 Day, are not necessarily related to IPv6.

Content NAT issues

One participant had issues with reachability over IPv6. When we contacted them, this was the explanation given: In their setup HTTP connections over IPv6 terminated to a device that NATted the source IPv6 addresses to a single IPv4 source. These NATted connections terminated on an IPv4-only service that implemented per source-IP rate-limiting. Adding to the problem was the fact that the web service was outsourced, and relatively hard to reach. When we were made aware of the problem, we reduced the number of connections that our measurement infrastructure made to this particular World IPv6 Day participant, so the per source-IP rate-limiting was less likely to be hit.

Lessons that can be learned from this:

• NAT can introduce unintended side effects, in pure-dual-stack this would not have been a problem.
• Have good Service Level Agreements with external parties you depend on.

Website reachability - US Department of Commerce

While no events were seen with the US Department of Commerce website on World IPv6 Day, there was a glitch in the aftermath. A little after 2:30AM UTC on 9 June, HTTP fetches over IPv6 as well as ping6-measurements stopped working, while the AAAA record for www.commerce.gov was still being returned in DNS. We don't see anything unusual in the traceroute6 data collected, nor anomalies in BGP. Around 14:30 UTC on June 9 the AAAA records are retracted, but due to DNS caching they stayed visible for hours. You can see this in Figure 3 which shows the success rate of DNS, ICMP and HTTP measurements. The HTTP over IPv6 and ping6 success rate is not 100% due to a few monitors that didn't have IPv6 connectivity to www.commerce.gov and some other monitored sites.

Figure 3: Success rate of DNS,ping and HTTP IPv6 measurements to www.commerce.gov

If people have details on what happened here, please comment below.

A lesson that can be learned from this:

• It is essential to have good monitoring, and channels for external parties to report problems.

Conclusion

Most of what we saw on World IPv6 Day were minor issues, that were caused more by making network changes, then with deploying IPv6. As network changes are done by humans, mistakes can and will be made, and in most cases technicians worked together to solve issues swiftly.

None of the glitches identified during World IPv6 Day are stumbling stones for production IPv6 deployment. So, if you haven't deployed it already, what's stopping you? :-)

If you have observed any other problems, please do not hesitate to describe them below, possibly together with some lessons learned.