RIPE Atlas has a large enough footprint all over the world to observe various kinds of network behaviour. Recently we've been notified about a case where the IPv6 source address ::/128 appeared in some of our traceroutes. We set out to discover why this happens.
RIPE Atlas collects active measurement data from 11,000+ probes deployed by community members in thousands of different networks. As part of their tasks these devices also collect information about the topology of the whole Internet by periodically executing traceroute measurements towards all the prefixes (IPV4 and IPv6) advertised in BGP (see note below). They also execute measurements specified by other RIPE Atlas users.
In a recent Tweet, the host of one of these probes, Daryll Swer, asked us why his probe's IPv6 traceroute results contain hops with source address ::/128. We did not have an immediate answer to this question but as this is unexpected behaviour we started to dig deeper. We used the open dataset of RIPE Atlas hosted in Google BigQuery to extract relevant information.
During a traceroute, the probe sends data packets to a destination address, while intentionally limiting the “hop count” value. The hop count indicates the number of routers the packet may pass before being discarded. When the hop count is exceeded, the router that received the packet will respond to the sender with the ICMP message “hop limit exceeded in transit”. It also attaches part of the offending packet. A capture of such a packet was provided by the host:
The original UDP packet was sent from
2607:fcb8:2::230. The router sends its response back to
2a10:6747:1:202:0:242:ac11:2, but with the source address of
The probe is unaware which routers its packets may pass on the way to their destination. Because of this and the stateless nature of ICMP, the probe cannot affect the address a router may use to respond with. This suggests the presence of at least one router in the path that responds in an unexpected manner. The question is where is this router?
Our initial findings
On 20 April 2022, RIPE Atlas collected about 85.7 million IPv6 traceroute results. Out of these 937,046 (roughly 1.09%) contained at least one hop responding with source address
::/128. These results came from 298 probes of about 11,250 connected on that day, so roughly 2.5% of all the connected probes. Curiously, this includes about 150 RIPE Atlas anchors, of which there are only 750 so these are way over-represented in the results!
|Probe version||Affected probes||Connected probes||Affected probes %|
No probe version seems to be immune to this effect. But anchors, as we said above, see it much more often! Why?
We also looked at where the affected probes are geolocated. The percentage of affected probes is really only representative if we have enough probes in a country, so the table below only shows those countries where there are at least 25 connected probes.
|Country code||Affected probes||Affected results||Total connected probes||Affected connected probes %|
Okay, we are definitely on to something here! Half of the probes in Kazakhstan (KZ) have seen this. In fact, when we order the full list of all affected probes by how many instances of the issue they recorded, the top 25 probes are all from the same country! In terms of total affected results, that is almost 86% of the observations, while the whole list of countries (of at least one affected probe) is 57 long…
Looking into the details of traceroutes
To further understand the source of
::/128 in the traceroutes, we mapped the IP addresses in affected traceroutes to their origin AS using the RIS data. The table below shows the number of traces where we had the same AS before and after
:: hop in the traceroutes. We suspect most of the affected probes in KZ are hosted in ASes which peer with AS9198 (JSC Kazakhtelecom).
|ASN||Country||No of Traces|
|9198 (JSC Kazakhtelecom)||KZ||131,937|
|TIMEOUT: no response||6,846|
|3701 (University of Oregon)||US||27|
|53087 (TELY Ltda.)||BR||9|
|61832 (Fortel Fortaleza Telecomunicacoes Ltda)||BR||2|
|6939 (Hurricane Electric LLC)||US||2|
|12389 (PJSC Rostelecom)||RU||1|
Moreover, we found that traceroutes from 14 out of 19 affected probes in the Netherlands also traverse through AS9198 in their path. The key takeaway here is that the probes topologically closer to the affected AS will also show up in the results. We are also investigating other ASes where we are unsure due to non-responsive IP addresses. We plan to reach out to providers where we have some level of confidence to discuss further steps.
Anchors are more often affected
Finally, to answer the question: why are anchors over-represented here?
RIPE Atlas anchors are special in a number of ways. Each one of them is targeted by a number of regular (non-anchor) probes continuously by measurements - in fact this is why they are called anchors: they provide stable “anchoring points” for the rest of the probes. More importantly for the current discussion, the anchors are also in a full measurement mesh, meaning that they target each other quite regularly with different kinds of measurements, including traceroutes.
Our strong suspicion is that this feature is the ultimate reason why we see such behaviour more often with anchors. Specifically, assuming there is really a network device in KZ responding with those packets, not only anchors “from KZ tracerouting outwards” see this, but also anchors outside of KZ targeting other anchors inside KZ have a higher chance of observing the issue. Since most regular probes do not target anchors in KZ, there are less cases on those.
Open data / open code
Virtually all RIPE Atlas measurement results are already available to the public. We are also publishing RIPE Atlas measurement results publicly on Google BigQuery. You can follow this page on RIPE Labs for more information. In this specific case, since only a small subset of the results contain the observed behaviour, we are sharing this subset in a separate download.
We are also publishing the queries we used to extract and process this dataset in Google BigQuery, with the intention that following this code, or adapting it to other scenarios, it’ll become easier for researchers and other interested parties to process data collected by RIPE Atlas and get to the information they need.
- 5051, 5151, 6052 and 6152, as described at https://github.com/RIPE-Atlas-Community/ripe-atlas-tips-and-tricks/wiki/Topology-Scanning
Vladimir Kaspirovich •
This scenario is caused by using 6PE in the Kazakhtelecom core network (AS9198) and it is an expected behavior . Kazakhtelecom core network does not use IPv6 addressing, and when using 6PE, IPv6 packets are tunneled over the IPv4/MPLS core, as a result of which zero values will be displayed during traces. If at least one IPv6 address is configured on the core device, for example, on a loopback interface, then the IPv6 address will be displayed during traces. This implementation scheme is a common practice, and is not some kind of erroneous scenario. https://www.youtube.com/watch?v=zraAfs3bY5E&t=377s.
Hide 2 replies
Daryll Swer •
I'm not very well familiarised with 6PE - But as I understand it and as per RFC6890 a packet where source IP = ::/128 cannot traverse inter-AS as it is not a routable address. So while it seems we now have an explanation for the ::/128 origin, the other questions is: Why packets with ::/128 are able to traverse inter-AS and into the DFZ?
Hide one reply
Daryll Swer •
The answer for inter-AS traversal is simple: Nobody is performing strict source address validation, and it's crazy they permit ::/128 to traverse in the first place! Aka lack of MANRS.
Anatoliy Kushner •
there replies from routers without any ipv6 address configured on it