Traceroute is a powerful tool for uncovering how packets travel across the Internet. However, as we recently discovered, some routers tamper with the TTL values in traceroute's probing packets, giving you a false view of how networks are actually interconnected.
This article was selected as winner of the RIPE Labs article competition for RIPE 93.
Traceroute is a powerful tool if you want to investigate how packets on the Internet get from source to destination. Many people use traceroute for debugging network issues or to explore the topology of the Internet. But do the paths and network links that we see in traceroute actually correspond to the ones on the Internet?
As we uncovered in a recent study, some routers can mess with your trace packets, giving you a false impression of what the network path actually looks like. To understand how this can happen, let’s start with a brief recap of how traceroute works under normal circumstances.
How traceroute is supposed to work
The goal of traceroute is to uncover the routers a packet traverses on its way to a target on the Internet. Traceroute does this by sending probing packets with increasing Time To Live (TTL) values to the destination. Usually, the routers along the path will decrease the TTL whenever they forward a packet. If the TTL reaches zero at a router, the router should drop the packet and respond with an ICMP Time Exceeded message.
By sending probing packets with increasing TTLs, we can therefore progressively uncover routers along the path:
But what happens if one of the routers along the path does not behave as expected? Let’s assume that R2 in the above example does not decrement the TTL, but overwrites the previous TTL with a fixed value, for example 255:
In this case, we can no longer elicit any ICMP error messages from the routers after R2: the packet will reach the destination once R2 changes the TTL to a higher value.
Now here’s the catch: If we do not know that R2 rewrites our TTL, we would infer that the destination and the router before R2 (R1) are directly connected. Depending on where such rewrites happen, we might therefore be missing entire networks in our traceroutes!
Does this really happen on the Internet?
At this point, you are probably asking: Does this really happen on the Internet? To investigate this, we ran traceroute measurements from RIPE Atlas probes to several destinations under our control where we could capture the traceroutes’ incoming probing packets. By capturing incoming traceroute probes, we can check whether the probing packets arrive with the expected TTL or a TTL higher than any of the TTLs sent in the course of the traceroute.
Indeed, among the Atlas Probes that we investigated, we found traceroutes from 950 probes (800 IPv4, 538 IPv6) in 471 ASes (446 IPv4, 263 IPv6) that were affected by rewrites, i.e., their trace packets arrived with TTL values that were too large.
By looking at the last hop that we observe in these traces, we can get an idea in which networks these traces are more likely happening. Surprisingly, TTL rewrites also happen in large Tier-1 providers: for example, we find 14 probes in Orange's Network (AS5511) that are affected by rewrites while the last visible hop is still in the source AS. Similarly, in AT&T's network (AS7018), we find 65 probes that experienced a rewrite while their trace probes were still within the network. There are, however, also probes where the rewrites don't happen in the source AS, but somewhere further down the path. A good example of this is Arelion (AS1299): we find that the traces from several ASes (4 IPv4, 16 IPv6) experience a rewrite after traversing this network.
How long have TTL rewrites been happening?
OK, so we know that these rewrites are happening, but how long has this phenomenon been around? RIPE Atlas has been running traceroute measurements for years - is there a way we can find out if any of these were already affected by rewrites in the past?
If we want to answer that question, we have to be able to tell with which TTL the probing packets arrived at the destination solely based on the data published in RIPE Atlas. Luckily, there is a method how we can infer just that. When a host responds with an ICMP error message, it also appends the header and the first 64 bits of the packet that elicited the error to the response (i.e., it "quotes" the offending packet so that the original sender can tell which message caused the error). Now, if one of our trace packets elicits, for example, an ICMP port unreachable message (which is often the case when we are doing a UDP traces), the error message from the destination includes the IP header of the final trace packet, which, among other things, contains the TTL - we can use this to get a glimpse at the TTL with which our probing packets arrived at the destination!
RIPE Atlas does occasionally save this “inner TTL” (iTTL) in the trace data - although this is an optional field, it still gives us a subset of traces where we can check if they were affected by rewrites.
So, we took a look at several snapshots of all the daily RIPE Atlas traces over the last couple of years that received an error from the destination and had an iTTL value stored. Again, we check for traces that arrive at the destination with a TTL higher than that of any of the probing packets. Turns out, TTL rewrites are by no means new: this sort of thing has been happening since at least 2018, and the absolute number of traces affected by rewrites has been increasing over time.
So what does this mean for traceroute users?
If you happen to see strange connections in traceroutes that, in your opinion, should not exist, it might very well be that these are the result of a router messing with the TTL values. Also, if you are using a platform like RIPE Atlas or CAIDA Ark to run large traceroute campaigns, you might want to double check if any of your vantage points are located in or traverse an AS that tampers with your TTL fields.
In the end, it is worth keeping in mind that traceroutes are not always accurate: just because traceroute shows a specific path or link does not mean that it has to actually exist on the Internet.
Comments 0