Measuring World IPv6 Day - Some Glitches And Lessons Learned

Emile Aben — Jun 28, 2011 10:25 AM
Filed under: , ,
As already stated by us and others, World IPv6 Day on 8 June, was a big success. No major issues came to light, and most minor issues that surfaced were resolved, and as such were a useful learning experience. In this article we provide details on some of the events we observed by looking at our measurement data.

The RIPE NCC measured World IPv6 Day in various ways as described in RIPE NCC Measurements for World IPv6 Day. One of the ways the RIPE NCC measured World IPv6 Day was to conduct active measurements from 40 vantage points to a selection of 53 World IPv6 Day participants. These measurements consisted of DNS,ICMP, HTTP and traceroute measurements both on IPv4 and IPv6. All measurement results are available on http://v6day.ripe.net. In this article we will look at some of the events we found in more detail.

HTTP 404 - Level3

When visiting the Level3 Communications website over an IPv6 connection, users didn't see the normal homepage. Instead they saw 404 document-not found error-pages over IPv6 (screenshot below), while connections over IPv4 got normal HTTP responses. This issue appeared in our data from 22:05 UTC on 7 June to 15:14 on 8 June. This was later acknowledged to be a webserver configuration mistake, that was unrelated to IPv6. While this issue was solved eventually, we were surprised to see it took several hours to be fixed.

Level3 website over IPv6

Figure 1: HTTP 404 Error as appeared on Level3 website when accessed over IPv6

 

A lesson that can be learned from this:

  • as with any technical infrastructure, it is essential to have good monitoring, and channels for external parties to report problems.

Partial reachability from vantage points

Some of our vantage points also had a problem reaching Level3 over IPv6 at all. This is due to some networks being partitioned, ie. if you are single-homed in one of these networks in IPv6 you can't reach the other. As can be seen in this Wikipedia article, the most notable cases of this seem to involve Cogent, but it also affects Hurricane Electric and, in this case, Level 3. Partitioning decreases the value of having one of these networks as your sole transit provider. It would be good for the networks involved to minimise this. Unfortunately partitioning is not new to IPv6, and this happens in IPv4 as well (over, and over, and over, and over again).

Three of our vantage points had problems reaching a majority of the participants we monitored over IPv6. For two of the CAIDA Ark vantage points, one in Manila, the other one in Dakar, these problems persisted. As these are both in educational institutions, the suspicion is that in IPv6 they are only able to reach other educational institutions through peering of NRENs (National Research and Education Networks). For TTM box tt120 there was a problem with the local network configuration that quickly got fixed once it got to the attention of the people operating the infrastructure.

A lesson that can be learned from this:

  • Be on the lookout for partial reachability. For instance, we could have done a better job in testing our vantage points for this, prior to World IPv6 Day.

Authoritative DNS issues - Juniper

For www.juniper.com an issue regarding DNS resolution was reported, and diagnosed as a problem with DNS glue records. Our measurement data shows this issue lasted from approximately 9:30 to 11:15 UTC on 8 June, and wasn't limited to missing glue records. As can be seen in Figure 2, both A and AAAA queries for www.juniper.com in this period resulted in a variety of responses. In about 60% of cases we saw a SERVFAIL returned to our vantage point both for A and AAAA. In about 30% of cases no DNS error code was returned, but also no resource record. The type of response was strongly correlated to the vantage point. About 10% of queries got responses with a resource record in it, possibly due to caching (we didn't record TTLs of returned resource records).

 

DNS queries to www.juniper.com

 Figure 2: Rate of DNS responses for A (IPv4) and AAAA (IPv6) for queries to www.juniper.com.

A lesson that can be learned from this:

  • Things going wrong on World IPv6 Day, are not necessarily related to IPv6.

Content NAT issues

One participant had issues with reachability over IPv6. When we contacted them, this was the explanation given: In their setup HTTP connections over IPv6 terminated to a device that NATted the source IPv6 addresses to a single IPv4 source. These NATted connections terminated on an IPv4-only service that implemented per source-IP rate-limiting. Adding to the problem was the fact that the web service was outsourced, and relatively hard to reach. When we were made aware of the problem, we reduced the number of connections that our measurement infrastructure made to this particular World IPv6 Day participant, so the per source-IP rate-limiting was less likely to be hit.

Lessons that can be learned from this:

  • NAT can introduce unintended side effects, in pure-dual-stack this would not have been a problem.
  • Have good Service Level Agreements with external parties you depend on.

Website reachability - US Department of Commerce

While no events were seen with the US Department of Commerce website on World IPv6 Day, there was a glitch in the aftermath. A little after 2:30AM UTC on 9 June, HTTP fetches over IPv6 as well as ping6-measurements stopped working, while the AAAA record for www.commerce.gov was still being returned in DNS. We don't see anything unusual in the traceroute6 data collected, nor anomalies in BGP. Around 14:30 UTC on June 9 the AAAA records are retracted, but due to DNS caching they stayed visible for hours. You can see this in Figure 3 which shows the success rate of DNS, ICMP and HTTP measurements. The HTTP over IPv6 and ping6 success rate is not 100% due to a few monitors that didn't have IPv6 connectivity to www.commerce.gov and some other monitored sites.

Measurements to US Department of Commerce

Figure 3: Success rate of DNS,ping and HTTP IPv6 measurements to www.commerce.gov

If people have details on what happened here, please comment below.

A lesson that can be learned from this:

  • It is essential to have good monitoring, and channels for external parties to report problems.

Conclusion

Most of what we saw on World IPv6 Day were minor issues, that were caused more by making network changes, then with deploying IPv6. As network changes are done by humans, mistakes can and will be made, and in most cases technicians worked together to solve issues swiftly.

None of the glitches identified during World IPv6 Day are stumbling stones for production IPv6 deployment. So, if you haven't deployed it already, what's stopping you? :-)

If you have observed any other problems, please do not hesitate to describe them below, possibly together with some lessons learned.

 

8 Comments

Marco Davids
Marco Davids says:
Jun 28, 2011 11:22 AM
I did a little bit of testing from the IPv6-only network and was somewhat disappointed to find out that certain websites had activated IPv6 for only a part of their website. In particular their CSS-content was often still served from an IPv4 box, which made their websites look pretty silly on IPv6 (only).

emileaben
emileaben says:
Jun 28, 2011 11:51 AM
Hi Marco,

This was indeed an issue for IPv6-only clients, but IPv6-only clients were not the specific target audience for World IPv6 Day. Maybe a next World IPv6 Day/Week/Month could have this as a specific goal.
JaapAkkerhuis
JaapAkkerhuis says:
Jun 28, 2011 11:38 AM
After June 8th I noticed I couldn't reach the L-root anymore via ipV6. At first I brushed it of as "it must be something local" but according to dnsmon the problem seems more widespread. And if one looks closely it actually gets worse over time. See <http://dnsmon.ripe.net/dns-[…]d=lastval&plot=SHOW> for details.

From my point of view, the packets seems not to survive occaid.net (Full traceroute on request.
emileaben
emileaben says:
Jun 28, 2011 12:20 PM
Hi Jaap,

Interesting. I'm not entirely sure it is World IPv6 Day related, since it started on 10 June, but good that World IPv6 Day made people monitor issues in IPv6 more closely.
Ted
Ted says:
Jun 29, 2011 08:10 PM
I've noticed *since* IPv6 day that enterthegame.com's DNS servers return A records to AAAA queries, resulting in name resolution failure with no IPv4-fallback. The trouble with the misbehaving nameservers is that (libc) considers it an error and does not follow up with an A query - which would have worked. I contacted the admin and he basically said that IPv6 is only for hobbyists and nobody uses it yet (despite that it breaks dual-stack users).

So some domains might be black-holing themselves to dual-stack users without realizing it. I doubt there are many DNS servers with that kind of misbehavior, but it is interesting to find out that enabling a working dual-stack environment for your users can result in inaccessible domains - and when the sysadmins refuse to accept that there is a problem, you have the option of leaving it broken or working around it for your users (which I did).
emileaben
emileaben says:
Jun 30, 2011 10:16 AM
Hi Ted,

While IPv6 usage is not ubiquitous, I don't think the 2% of visitors of www.ripe.net that use IPv6 would agree with that admin. ;)

I'm not aware of any misbehaviour of nameservers that you describe, so I'd be curious to find out more about it. Do you have tcpdumps of misformed packets and/or DNS-server version, or any other info that could help recreate the problem, so it can be assessed what the global impact of this type of misbehaviour is?

(you can mail us at labs@ripe.net as well, if you don't want to comment publicly)
Colin
Colin says:
Jul 05, 2011 10:22 AM
Work: BTNet - no IPv6 support, public statement gives 2012 for rollout
Home: TalkTalk - no IPv6 support. Not yet found a source that says anything apart from "IPv4 will be with us for 5-10 years yet so we don't need to do anything just yet"
Web server (big USA provider): "We have lots of IPv4 addresses. Don't worry, by the time you need it we'll have it ready"
If I could use native IPv6 at these locations today would I?
Work: Absolutely!
Home: If it was there I would make sure I was set up for dual stack
Web server: Absolutely!
No tunnels please, let's just get on and do this properly.
emileaben
emileaben says:
Jul 05, 2011 11:47 AM
Hi Colin,

Yes, some parties are behind. The call for deployment in the article was mostly addressed to ISPs.

Maybe you can have the folks that are not convinced of the need for IPv6 read RIPE Labs more often ;)
Add comment

You can add a comment by filling out the form below. Only plain text is possible. Web and email addresses will be transformed into clickable links. Comments are moderated so they won't appear immediately.

Navigation
Related Items
Increased Reach of RIPE Atlas Anchors

Increasing the reach of RIPE Atlas anchors is one of the highest priority goals of RIPE Atlas Team. ...

Proposing Making RIPE Atlas Data More Public

RIPE Atlas is now three years old, and is moving from a prototype to production service. Based on ...

Modifications to the IP Analyser to Reflect New Policy

We are in the process of implementing the policy regarding Post Depletion Adjustment of Procedures ...

Report on IPv6 Security Test Methodology

The Dutch Institute for Applied Scientific Research (TNO) and a number of Dutch security companies ...

RIPE Atlas: Improved Probe Pages

We've made it much easier to get an overview of the history and measurements for all the public ...

more ...