The Curious Case of Packets From ::

RIPE Atlas collects active measurement data from 11,000+ probes deployed by community members in thousands of different networks. As part of their tasks these devices also collect information about the topology of the whole Internet by periodically executing traceroute measurements towards all the prefixes (IPV4 and IPv6) advertised in BGP (see note below). They also execute measurements specified by other RIPE Atlas users.

In a recent Tweet, the host of one of these probes, Daryll Swer, asked us why his probe's IPv6 traceroute results contain hops with source address ::/128. We did not have an immediate answer to this question but as this is unexpected behaviour we started to dig deeper. We used the open dataset of RIPE Atlas hosted in Google BigQuery to extract relevant information.

The problem

During a traceroute, the probe sends data packets to a destination address, while intentionally limiting the “hop count” value. The hop count indicates the number of routers the packet may pass before being discarded. When the hop count is exceeded, the router that received the packet will respond to the sender with the ICMP message “hop limit exceeded in transit”. It also attaches part of the offending packet. A capture of such a packet was provided by the host:

The original UDP packet was sent from 2a10:6747:1:202:0:242:ac11:2 to 2607:fcb8:2::230. The router sends its response back to 2a10:6747:1:202:0:242:ac11:2, but with the source address of ::.

The probe is unaware which routers its packets may pass on the way to their destination. Because of this and the stateless nature of ICMP, the probe cannot affect the address a router may use to respond with. This suggests the presence of at least one router in the path that responds in an unexpected manner. The question is where is this router?

Our initial findings

On 20 April 2022, RIPE Atlas collected about 85.7 million IPv6 traceroute results. Out of these 937,046 (roughly 1.09%) contained at least one hop responding with source address ::/128. These results came from 298 probes of about 11,250 connected on that day, so roughly 2.5% of all the connected probes. Curiously, this includes about 150 RIPE Atlas anchors, of which there are only 750 so these are way over-represented in the results!

No probe version seems to be immune to this effect. But anchors, as we said above, see it much more often! Why?

We also looked at where the affected probes are geolocated. The percentage of affected probes is really only representative if we have enough probes in a country, so the table below only shows those countries where there are at least 25 connected probes.

Okay, we are definitely on to something here! Half of the probes in Kazakhstan (KZ) have seen this. In fact, when we order the full list of all affected probes by how many instances of the issue they recorded, the top 25 probes are all from the same country! In terms of total affected results, that is almost 86% of the observations, while the whole list of countries (of at least one affected probe) is 57 long…

Looking into the details of traceroutes

To further understand the source of ::/128 in the traceroutes, we mapped the IP addresses in affected traceroutes to their origin AS using the RIS data. The table below shows the number of traces where we had the same AS before and after :: hop in the traceroutes. We suspect most of the affected probes in KZ are hosted in ASes which peer with AS9198 (JSC Kazakhtelecom).

Moreover, we found that traceroutes from 14 out of 19 affected probes in the Netherlands also traverse through AS9198 in their path. The key takeaway here is that the probes topologically closer to the affected AS will also show up in the results. We are also investigating other ASes where we are unsure due to non-responsive IP addresses. We plan to reach out to providers where we have some level of confidence to discuss further steps.

Anchors are more often affected

Finally, to answer the question: why are anchors over-represented here?

RIPE Atlas anchors are special in a number of ways. Each one of them is targeted by a number of regular (non-anchor) probes continuously by measurements - in fact this is why they are called anchors: they provide stable “anchoring points” for the rest of the probes. More importantly for the current discussion, the anchors are also in a full measurement mesh, meaning that they target each other quite regularly with different kinds of measurements, including traceroutes.

Our strong suspicion is that this feature is the ultimate reason why we see such behaviour more often with anchors. Specifically, assuming there is really a network device in KZ responding with those packets, not only anchors “from KZ tracerouting outwards” see this, but also anchors outside of KZ targeting other anchors inside KZ have a higher chance of observing the issue. Since most regular probes do not target anchors in KZ, there are less cases on those.

Open data / open code

Virtually all RIPE Atlas measurement results are already available to the public. We are also publishing RIPE Atlas measurement results publicly on Google BigQuery. You can follow this page on RIPE Labs for more information. In this specific case, since only a small subset of the results contain the observed behaviour, we are sharing this subset in a separate download.

We are also publishing the queries we used to extract and process this dataset in Google BigQuery, with the intention that following this code, or adapting it to other scenarios, it’ll become easier for researchers and other interested parties to process data collected by RIPE Atlas and get to the information they need.

Note:

• 5051, 5151, 6052 and 6152, as described at https://github.com/RIPE-Atlas-Community/ripe-atlas-tips-and-tricks/wiki/Topology-Scanning