The Shape of a BGP Update

Vasco Asturiano — Feb 14, 2011 04:00 PM
Filed under: ,
What happens to BGP traffic when an announcement or withdrawal of an address prefix propagates through the Internet? See below some interesting graphs and observations.

Introduction

What happens when an IP address prefix gets announced or withdrawn. How does this information propagate through the Internet? And how does it affect the amount of BGP traffic across the Internet when a single prefix is freshly announced or withdrawn from the global routing table? This is not going to be surprising for any BGP expert. The intent of this story is to provide some general intuition about the current state of BGP routing in practice.

Methodology

The prefix 84.205.67.0/24 was not visible in the routing table on 2 February 2011 at 08:00 UTC. At that point it was announced in BGP to all 61 of our local directly connected peers via our Routing Information Service (RIS) Route Collector present at the Amsterdam Internet Exchange AMS-IX.

Two hours later, at 10:00 UTC, the same prefix was actively withdrawn via the same set of peers at AMS-IX.

We measured the BGP update traffic that was generated by these two events from all of the RIS collectors by observing the incoming updates via 90 full table peers across the globe.

The prefix 84.205.67.0/24 is part of the set of RIS beacons that have periodical and pre-determined patterns of announcements and withdrawals. Find more at: http://www.ripe.net/data-tools/stats/ris/current-ris-routing-beacons

Results

For the prefix announcement occurring at 08:00 UTC, Figure 1 below represents the update activity (BGP updates per second) observed over time, as well as the visibility progress of the prefix (the number of peers seeing it).

The 0 seconds at the X axis marks the moment the prefix got announced.

The visibility is calculated by dividing the number of peers already seeing the prefix by all the peers in the set (90).

For the BGP updates we differentiate between (1) the first update ever sent by a peer regarding this event and (2) further updates from a peer that has already seen the event. This allows us to distinguish between updates that represent the first time they are seen by a peer, and updates that most likely represent route selection (AS path convergence, etc).

Prefix Announcement – BGP Updates  and Propagation of Visibility

Figure 1: Prefix Announcement - BGP Updates and Propagation of Visibility

We observe that all of the generated BGP traffic for this announcement occurs in slightly over a minute after the initial prefix announcement, with 90% of the volume occurring in the first 30 seconds. Full visibility (100%) by all peers has been reached after only 38 seconds. A visibility of 40% is accomplished in just a few seconds.

Nearly half of the updates (46%) are further updates from peers, representing the overhead of the BGP convergence.

Figure 2 below shows the BGP update volume for the prefix withdrawal that occurred at 10:00 UTC. Here we distinguish between the update messages per type: announcement or withdrawal. For visual clarity, withdrawals are represented on the negative side of the Y axis. As before, both types are separated between first and further updates as seen by a peer.

Prefix Withdrawal – BGP Updates

Figure 2: Prefix Withdrawal - BGP Updates and Propagation of Visibility

It is interesting to note that the generated volume of traffic for the prefix withdrawal is much higher than for the announcement of the same prefix. Almost 4 times more BGP update messages (503 versus 139) are registered. Also, the activity lasts for nearly 3 minutes as compared to just over a minute for the prefix announcement.

79% of all updates are further updates from a peer. This indicates that on average a route withdrawal will generate about 5 BGP messages sent by a peer, as compared to only 1 or 2 messages for a route announcement.

Also, only 25% of the messages are withdrawal messages, indicating that a peer sends an average of 3 route re-announcements before withdrawing it. Furthermore, most of the withdrawal messages are concentrated at later than 1min after the the initial prefix withdrawal (when visibility is still 80%), with an accented burst at around 90 seconds when visibility drops from 60% to 20%, indicating the moment when the withdrawal of the prefix from the global routing tables starts converging more abruptly.

Note that only 2.6% of the update messages will be concerning a peer that receives the withdrawal for the first time and immediately passes it on to its peers.

The distribution of the BGP traffic per update message type for the whole period is shown below in Figure 3.

BGP Withdrawal - Update Type DistributionFigure 3: Prefix Withdrawal - Update Type Distribution

This phenomenon of the much larger volume of updates, with multiple update messages from each peer can be explained by the way routers interpret a route withdrawal: A router has no way of knowing if a route withdrawal is happening because it was withdrawn at the origin, or because its peer simply dropped the route due to connectivity loss or other malfunction. Therefore, following the resilient nature of the Internet, it will look in its routing table for an alternative path towards the withdrawn prefix and try to save the connection to the prefix in that way. If it succeeds, it will pass the update to its peers, therefore prolonging the life of the route. Only when it runs out of alternative paths, will it finally remove the prefix from its routing table and send the withdrawal message to its peers.

Conclusion

To conclude, we observe that BGP route updates tend to converge globally in just a few minutes. The propagation of newly announced prefixes happens almost instantaneously, reaching 50% visibility in just under 10 seconds, revealing a highly responsive global system. Prefix withdrawals take longer to converge and generate nearly 4 times more BGP traffic, with the visibility dropping below 10% only after approximately 2 minutes.

Keeping all this in mind, we can conclude that it is often easier and quicker to turn on a prefix, than to switch it off.

6 Comments

Iñigo Ortiz de Urbina
Iñigo Ortiz de Urbina says:
Feb 24, 2011 02:58 PM
It made me wonder if there is any significant difference between the announced IPv4 and IPv6 beacons. Any plans on repeating the analysis and publishing the results for any of the IPv6 beacons?

Thanks in advance!
vastur
vastur says:
Mar 11, 2011 04:46 PM
Hi Iñigo,

Thanks for your suggestion. We didn't look deeply into the behavior of IPv6 prefixes, but they show a similar pattern of updates when announcing or withdrawing, with perhaps a slightly shorter convergence time. No other remarkable differences were noticed at first glance but a deeper analysis might reveal further insight.
We'll consider a follow up on this subject to run a comparison analysis of how the behavior compares with IPv4.

Thanks,
Vasco
Eric Miller
Eric Miller says:
Feb 20, 2013 02:55 AM
I just wanted to thank you for posting this article. You provided extremely useful information that is hard, if not impossible, to find.

Thank you!

Eric
Eric Miller
Eric Miller says:
Feb 21, 2013 03:03 AM
Question - do you anticipate the same convergence timing if a route is withdrawn from one ASN and subsequently advertised on a different ASN?

Eric
Vasco Asturiano
Vasco Asturiano says:
Feb 22, 2013 02:37 PM
Hi Eric,

Thanks for your comment.

We haven't investigated that specific case, therefore we can only speculate. If the announcement of the second ASn would happen after the prefix was completely withdrawn, then most likely the model would be the same as portrayed above.
However, if the second ASn announcement takes place before the withdrawal process finishes, one could expect some degree of cross-influencing, and would not be surprising to find that the total number of BGP messages would be less. And probably a shorter convergence time. How much, remains to be measured in a real-case experiment.

Thanks,
Vasco
Eric Miller
Eric Miller says:
Feb 23, 2013 09:30 PM
Thanks Vasco! I just wanted to let you know that we performed this procedure yesterday by withdrawing the route with the old ASN, waiting 5 minutes, then announcing the route with the new ASN, and the BGP updates were received throughout the Internet almost immediately. The withdrawal was seen within about 25 seconds at all locations we checked, some within 5 seconds. The announcement was just as fast, if not faster.

So, it appears that the convergence time was extremely fast and we had very little disruption. We probably could have avoided waiting the 5 minutes, but we wanted to be sure all routers had withdrawn the route prior to advertising. If we had not waited 5 minutes, the entire process could have taken less than 1 minute. Not bad!

Eric
Add comment

You can add a comment by filling out the form below. Only plain text is possible. Web and email addresses will be transformed into clickable links. Comments are moderated so they won't appear immediately.

Related Items
Increased Reach of RIPE Atlas Anchors

Increasing the reach of RIPE Atlas anchors is one of the highest priority goals of RIPE Atlas Team. ...

Proposing Making RIPE Atlas Data More Public

RIPE Atlas is now three years old, and is moving from a prototype to production service. Based on ...

RIPE Atlas: Improved Probe Pages

We've made it much easier to get an overview of the history and measurements for all the public ...

Visualising Bandwidth Capacity and Network Activity in RIPEstat Using M-Lab Data

As a result of the cooperation between the RIPE NCC and Measurement Lab (M-Lab), you can now ...

Internet Disruptions in Sudan

Significant Internet disruptions are happening in Sudan, possibly as a reaction to riots. We use ...

more ...