Hampering Eyeballs - Observations on Two "Happy Eyeballs" Implementations

IPv6 deployment is hampered due to end users that have broken IPv6 connectivity. Luckily this only affects an ever decreasing fraction of end users, but the fact that small amounts of broken IPv6 connectivity exists makes some people not deploy IPv6. When a user who is dual-stacked connects to a dual-stack website, his or her system (operating system plus application) will traditionally try to connect over IPv6 first (as per RFC 3484 ). When IPv6 doesn't work, this connection has to time out, before IPv4 will be attempted. Since timing out happens in the order of tens of seconds, the resulting user experience is horrible: Users end up staring at a blank screen or spinning wheels and will likely give up before the timeout occurs.

The selection of what IP address to connect to (IPv6 or IPv4) is typically done by picking the first result from the getaddrinfo() system call (ie. something that is baked into an operating system), and when that first result doesn't respond the next result from getaddrinfo() is tried. RFC 3484 defines in what order types of addresses should be tried. Roughly, RFC 3484 dictates that native IPv6 be tried first, then IPv4, and then IPv6-through-transition-technologies.

The "Happy Eyeballs" concept aims to address the situation of bad performance when a user encounters broken IPv6. "Happy Eyeballs" is a description of implementations of operating systems and/or applications that take measures so users ("eyeballs") won't see noticeable performance degradation when encountering broken connectivity in one protocol but not the other. Analogous one could call the users that do encounter the slow fall-back behaviour to broken dual-stack hosts using classic RFC 3484 "Unhappy Eyeballs". The term "Happy Eyeballs" itself was first coined in an IETF Internet Draft describing a specific implementation, which has evolved to describe the general principles of the mechanisms involved. See below a quote from the "Happy Eyeballs" draft:

A Happy Eyeballs algorithm has two primary goals:
1. Provides fast connection for users, by quickly attempting to connect using IPv6 and (if that connection attempt is not quickly successful) to connect using IPv4.
2. Avoids thrashing the network, by not (always) making simultaneous connection attempts on both IPv6 and IPv4.

There have been at least two different flavours of "Happy Eyeballs" deployed in commonly used operating systems and web browsers in the last few months. Below I will detail some of the behaviour I've seen from these two different flavours.

Chrome

The first "Happy Eyeballs" implementation that I'm aware of that made it into production software is included in the Chrome webbrowser since version 11.0.696.71 which was released on 24 May 2011. In Chrome's issue tracker the way getaddrinfo() -results are handled is described as follows:

When a hostname has both IPv6 and IPv4 addresses, and the IPv6 address is listed first, we start a timer (300ms) [...]. If the timer fires, that means the IPv6 connect() hasn't completed yet, and we start a second socket connect() where we give it the same AddressList, except we move all IPv6 addresses that are in front of the first IPv4 address to the end. That way, we will use the first IPv4 address. We will race these two connect()s and pass the first one to complete to ConnectJob::set_socket()

In real life this works pretty well. I've put a web server on an intentionally broken dual-stack hostname ( http://intentionally.broken.dualstack.wdm.sg.ripe.net/ , which redirects to www.ripe.net when accessed on IPv4 and has an IPv6 address that leads nowhere). Anytime I visited that intentionally broken dual-stack host using a recent version of Chrome, the fallback from the intentionally broken IPv6 to IPv4 was not noticeable to me.

The Chrome approach is simple but effective: If your IPv6 isn't fast enough (or not working at all), it's not going to be used. Period.

Mac OS X Lion

The second "Happy Eyeballs" implementation is included in the Mac OS X Lion operating system, released on 20 July 2011. In a post by an Apple engineer on the Apple IPv6 mailinglist, it is described in two parts. The first part is about getaddrinfo() :

Results from getaddrinfo are now sorted using routing statistics (destination
with the lowest min round trip time wins). If the statistics can not determine
which destination is better, an implementation of RFC3484 is used. The default
RFC3484 policy is read only.

The second part is about two Apple frameworks:

CF and NS layer frameworks that use CFSocketStream do not use getaddrinfo.
Those APIs use something similar to happy eyeballs. The A and AAAA queries are
started at the same time but the responses are handled as they are received.
When an answer is received, it is sorted in to a list of destination addresses.
If there are no more addresses coming in (this was the last answer in the DNS
packet or mDNSResponder has no more answers in the cache), a connection is
started to the first destination on the sorted list. The DNS resolve operation
is left running and more answers are processed as they arrive. A timer is setup
for a period of time in which we would expect the connection to complete, based
on the routing statistics. If the timer fires before the connection is
established, a connection to the next best address will be started while the
existing connection continues to try and make progress. A similar timer is
setup and the process repeats until a connection is established or we run out
of addresses to try. The code keeps track of whether or not it has received
both A and AAAA response (whether the answer was a list of addresses or no
address). If the connection is established before both A and AAAA responses
come back, the resolve is kept open for up to a second to allow mDNSResponder
to receive a slow response and store it in the cache. This way, subsequent
connections to the same host in a short period of time will have all answers in
the cache.

I also tested this to my intentionally broken dual-stack hostname, and the results are, well, complicated, so bear with me.

On Lion with Firefox 7 one has to wait for over 20 seconds before failover between intentionally broken IPv6 to IPv4 happens. My guess is that initially there are no routing statistics, so RFC 3484 is used and my Firefox waits 20 seconds in vain for a non-working IPv6, before falling back to IPv4. So Mac OS X Lion doesn't solve the broken dual-stack problem in this case. Firefox 7 does have a "Happy Eyeballs" mechanism, but it is disabled by default (which is what I tested). This default has not changed in Firefox 8, which I haven't extensively tested yet.

On Lion, Chrome's "Happy Eyeballs" saves the day: the 300ms fallback-mechanism as described in the previous paragraph about Chrome just works. If the Firefox 7 "Happy Eyeballs" mechanism is activated, it works as well as Chrome. This is done by setting the value of network.http.fast-fallback-to-IPv4 in about:config to true.

But Lion with Safari is further different. I'm guessing because it doesn't use getaddrinfo() for this, but one of the two frameworks described in the post above. A tcpdump reveals that indeed both connections, one IPv6, the other IPv4, are tried at the same time. And since the IPv6 in my test was intentionally broken, the IPv4 connection wins.

When visiting the broken dual-stack website first with Safari, and then with Firefox 7 things are different again. In this case Firefox 7 doesn't have the noticeable performance degradation, like it has without first using Safari. (Note: You may have to try a couple of times in Safari and wait a couple of minutes for the operating system to detect that IPv6 is not working before the Firefox 7 performance degradation due to slow fail-over goes away). This may seem weird  but makes sense if the routing statistics of both IPv4 and IPv6 connection attempts by Safari have been cached, and getaddrinfo() will return a list with the fastest protocol first. Because IPv6 failed, in this case this will be IPv4.

Are you confused yet? It looks like OS X Lion makes debugging connectivity issues to dual-stacked websites a royal pain, with the associated cost for support staff. While the Mac OS X Lion + Safari combination looks to be a true "Happy Eyeballs" implementation, the combination of OS X Lion with anything that just uses getaddrinfo() does not make the eyeballs much happier it seems.

Debugging Dual-Stack On Mac OS X Lion

I'm not aware of any tools to see what, and in what order, getaddrinfo() in Mac OS X Lion actually returns IP addresses. This matters because this can be different on every host due to differences in performance statistics and how much of this is cached. The post on the Apple IPv6 mailing list mentions nettop -n -m route to dump a live view of the routing statistics. However, this doesn't help you to match up the IP addresses that are associated with a hostname and find out which one was seen as faster easily.

Luckily it's easy to code up a tool for this or if you're lazy like me you can even find existing code online that will do the job. This will help in debugging issues with Mac OS X Lion. If people are aware of other useful tools here (bonus points if they are pre-installed on Mac OS X Lion), please comment below.

Windows 7

Windows 7 doesn't have a "Happy Eyeballs" implementation that I'm aware of. I tested both Firefox 8 and the latest Internet Explorer 9 beta against the intentionally broken dual-stack website, and in both cases encountered a long timeout before fall-back to IPv4.

The Effect of "Happy Eyeballs" on IPv6 deployment Metrics

The RIPE NCC runs a measurement infrastructure to monitor deployment of IPv6 at end-users, as we described on RIPE Labs earlier . Since we have over three months of data for both "Happy Eyeballs" implementations described above, we can see what the effects are on IPv6 traffic levels, and IPv6 deployment statistics that are coming out of infrastructures like ours (for instance Google 's, or the one that APNIC is running and the RIPE NCC is actively supporting).

With our measurement setup we can measure the effect of both "Happy Eyeballs" implementations on communication between dual-stacked hosts. In the the old non-"Happy Eyeballs"-world two native dual-stacked hosts would always try to communicate over IPv6 first. In the new, not necessarily better, world, this is not necessarily the case.

In our measurements, hosts will try to fetch objects that are available in IPv4-only, IPv6-only and objects that are available over both (ie. dual-stacked). I looked at the hosts that are able to fetch an IPv6-only object using unicast IPv6 (determined by having a non-Teredo, non-6to4 IPv6 source address). For these native-IPv6 capable hosts I looked at what percentage would fetch the dual-stacked object over IPv4. This is the tell-tale sign of "Happy Eyeballs" or other deviations from RFC 3484 at work, determining that IPv4 should be used instead of IPv6 for connections from that host to our measurement server. Let's call this behaviour "Hampering Eyeballs", since these hosts are using the legacy IPv4 protocol, while they have perfectly fine native IPv6 available, which is a pretty good approximation for this dictionary entry I found:

 hamper:
hinder or impede the movement or progress of

Table 1: Percentage of "Hampered Eyeballs" for Mac OS X Snow Leopard, Mac OS X Lion, and all other operating systems. The numbers in brackets are the minimum number of observations per month in this category.

As you can see in Table 1, the latest two versions of Mac OS X show quite some different behaviour. Mac OS X Snow Leopard shows a very good track record, while almost half of the clients using Mac OS X Lion are "Hampering Eyeballs", and the fact that they have perfectly fine working native IPv6 connectivity is masked. Remember that we excluded hosts with non-working native IPv6 connectivity from these numbers.

Table 2: Percentage of "Hampered Eyeballs" for pre- and post-Happy Eyeballs implementation in Chrome, as compared to other browsers. The numbers in brackets are the minimum number of observations per month in this category.

As you can see in Table 2, the "Happy Eyeballs" versions of Chrome hardly have a noticeable effect on what percentage of hosts was seen with the "Hampering Eyeballs" behaviour, as compared to older versions of Chrome, or other browsers. So implementing "Happy Eyeballs" the Chrome-way doesn't cause significant IPv6 capacity to be masked. At least not to hosts that have comparable IPv4 and IPv6 performance, like our measurement host. Since in Table 2 there is no distinction between operating systems, it is quite likely that OS X Snow Leopard causes some (most?) of the "Hampering Eyeballs" observed in all three categories.

Table 3: Percentage of "Hampered Eyeballs" for Lion + Safari, as compared to Lion + other browsers, or other combinations of operating system and browser. The numbers in brackets are the minimum number of observations per month in this category.

In Table 3, the combination of OS X Lion + Safari is compared to OS X Lion with other browsers and you can see that there is a difference. When using Lion + Safari, roughly half of the hosts measured, show the prefer-IPv4-to-reach-dualstack behaviour. What I expected to see is that Lion + other browsers would be as low as the category "Other combinations". Remember that getaddrinfo() on Mac OS X still uses RFC 3484 when no performance statistics about the destination IP address are known. This can be caused by hosts having performance statistics in the cache for the measurement host that we use to perform the IPv4, IPv6 and dual-stack tests, or there are other browsers that don't use getaddrinfo() anymore.

Conclusion

If the way Mac OS X implemented "Happy Eyeballs"-like functionality becomes more widespread (please don't do this Microsoft!) it will cause perfectly fine dual-stack hosts with roughly equal performance over both protocols, to select the legacy IPv4 protocol in roughly half of the cases. This makes the IPv6 capacity in networks under-used and that is bad for a  few reasons:

• There will be less incentive to improve the IPv6 network further (performance, more interconnects)
• It will put more strain on the IPv4 network, including middleboxes that are put in this network
• IPv6 deployment statistics will under-report IPv6 capacity, which will make informed decisions based on these harder

Prolonging the transition to IPv6 unnecessarily long (ie. causing "Hampering Eyeballs") is not going to help the Internet.

The Chrome implementation of "Happy Eyeballs" on the other hand seems to adequately avoid a degraded user experience ("Unhappy Eyeballs") when visiting broken dual-stacked hosts, and at the same time doesn't cause "Hampering Eyeballs" in significant numbers.

If people have other observations on "Happy Eyeballs", please comment below.

Note: After a few comments from readers I decided to change the technically incorrect "Hampered Eyeballs" (as the users are not hampered in viewing web pages), into "Hampering Eyeballs" (as the users are, inadvertently, hampering the transition to IPv6).