You are here: Home > Publications > RIPE Labs > Berislav Todorovic > The 1M Day Isn't That Far Away

The 1M Day Isn't That Far Away

Berislav Todorovic — 05 May 2020
In less than two years from now, the number of IPv4 prefixes in the global BGP routing table will hit the one million figure. Should you be worried? Most probably not, but better safe than sorry.

Back in August 2014, we witnessed a series of Internet "hiccups" worldwide when the number of IPv4 prefixes in the global BGP routing table hit the 512K boundary. A nice RIPE Labs article, along with references mentioned in there, explained the reasons behind this and illustrated the effects from the RIPE RIS point of view.

Last year, when the IPv4 prefix count reached 768K, there was a moderate concern that a similar issue could occur. Luckily, it turned to be too much ado about nothing, as explained in another RIPE Labs article

However, in two years from now, probably somewhere around June 2022 (+/- 6 months) we'll be facing another Internet milestone - one million IPv4 prefixes in the global routing table, according to the extrapolated BGP stats obtained from bgp.potaroo.net. Look at the graph below, created using the data from 2019. As you can see, the trend line is still valid. And while some people may argue about whether 1M is precisely 1,000,000 or 1,048,576, the difference is not that huge. What is so special about this? And why should you care?

Over the last decade, many vendors performed stress-tests on their routers and switches by generating 1M IPv4 prefixes. Upon successful tests most of them marketed their products with the statements like "Certified for 1M routes". 1M seemed to be enough, even though some routing platforms could process more than that. However, most of those routers were designed to store and process at least 1M prefixes. Of course, many routers nowadays can take much more, the order of magnitude in the industry is currently around 10M, so if your equipment is less than ~7 years old, you should probably forget all of this.

Still, better safe than sorry! And the check is usually simple: create an inventory of all the routing devices in your network running the full BGP global routing table and then ask your vendor to give you a clear answer about the maximum number of IPv4 and IPv6 prefixes those routers can take. You should pay attention to two different figures:

  • Control plane prefix capacity (a.k.a. RIB max prefix count): The number of routes that your network processor module (usually called RSP, CPM, Supervisor board, Routing Engine...) can store and process. Routing Information Base (RIB) is a copy of the routing table stored on the network processor module, used by various routing protocols (BGP, OSPF, IS-IS etc.). The limiting factor here is the amount of RAM on your network processor board. This is mostly not an issue nowadays, since modern network processors can easily handle more than 10M prefixes.

  • Forwarding plane prefix capacity (a.k.a. FIB max prefix count) - The number of routes that can be stored in your router's Forwarding Information Base (FIB). The FIB is being used by the router to decide on which egress interfaces to push the received IP packet, based on its destination address. Depending on your router architecture, the FIB may be stored either on the router linecards or in some kind of a centralised switching component. In either case, modern routers use TCAM to store IP prefixes, so the TCAM size determines the prefix count. Most of the times, this is the limiting factor, especially for older devices.

If you have the time and resources, you should definitely test those limits yourself. If you are lucky to have a separate test lab or at least one test device of the same model/type used within your live network, you should generate at least 1.2M BGP routes and see how your box behaves. You don't need expensive testers (e.g. Ixia, Spirent etc....) for that. There was a good article on ExaBGP (also here) on RIPE Labs. NOTE - the code was migrated from Google Code to a GitHub, see ExaNetworks exabgp repository. You can install ExaBGP on a Linux server and use it as a test generator. ExaBGP has a very nice add-on 'super-smash-brogp, allowing you to generate an arbitrary amount of IPv4 and IPv6 prefixes. Just set it to generate 1.2M. Observe the results, check RAM and TCAM utilisations, check router logs to see any signs of rejected prefixes. Contact your vendor if you see serious issues and start making mitigation plans.

Do not wait for bad things to happen. Be proactive!

2 Comments

Kurt Kayser says:
11 May, 2020 09:58 AM
Hello,
great article and I love the topic too.
An I really would like to know:
1. What is the total count of routers in the world that carry the full routing-table? My guess is: 10.000+
2. How many routers are affected to be upgraded between now and the 1Mio destinations in 2022?
3. I wonder if it could be (roughly) calculated how much one single entry currently costs in the global table?

regards, Kurt
Berislav Todorovic says:
12 May, 2020 10:54 AM
Hi Kurt,
Interesting questions, real thinking experiments. :) Let's think about it together.

1. Assume the classical 80/20 rule here. According to bgp.potaroo.net, there are ~70,000 ASNs in use worldwide at the moment, 25,000 of those advertising a single prefix. We can freely assume 80% of small networks don't use the full GRT, 80% of the large ones do. This gives 61,000 networks using GRT. Let's round this up to 60,000. Networks can have anything from one single box up to 10,000 routers (large Tier-1s) and, as a rule of thumb, most routers in large carrier networks do have the full BGP GRT. So, let's assume 500 edge (BGP-speaking) routers per network on average. This gives ~30M BGP routers worldwide. Much more than 10000 in any case. :)

2. As said in the article, most modern high-end routers (manufactured after 2012 at least) can handle way more than 1M prefixes in the FIB. So the chance for someone to still have a 10+ years old box connected to an Internet Exchange or having full GRT is small. Especially after the 512K hiccups, when everybody hopefully learnt their lessons. The 768K event had near to zero impact, so chances are great that everyone is more proactive these days, so we won't have large impact this time either.

3. This one is difficult to estimate, really. RIB uses RAM, which is normally ~1M/prefix per single copy of the GRT in the BGP RIB set (Adj-RIB-in + Loc-RIB + Adj-RIB-out), but RAM is typically not an issue. Even older network processors can take more than a single copy of the GRT on it and are scaled to 4M+ prefixes. The issue is in the FIB, which typically uses TCAM and TCAM stores prefixes in a tree-like fashion, so its utilization depends on the route aggregation efficiency and amount of overlapped prefixes. Each ASIC vendor has its own prefix store and lookup algorithm and this information is mostly restricted and/or protected as patents.

Regards,
Beri
Add comment

You can add a comment by filling out the form below. Comments are moderated so they won't appear immediately. If you have a RIPE NCC Access account, we would like you to log in.