More Leaky Routes

This time the problem started in Malaysia and its major Internet service provider in the country, Telekom Malaysia, on 12 June.

What Happened?

Now let’s go back to 12 June and look at what happened. Certainly something happened, as can be seen in the following two figures. The first is an hour-by-hour plot of the total number of prefixes announced in the Internet’s routing table.

Figure 1: Routing Table Size: Week starting 9th June 2015 (www.potaroo.net)

It looks as if 2,500 additional routes were added to the routing system, and then a couple of hours later some 5,500 routes were withdrawn, followed by the restoration of 2,500 routes, leaving the routing system some 500 routers smaller than the original state.

A similar picture is evident when looking at the updates in BGP across the same period, where the change in the BGP table size was matched by a corresponding burst in BGP update activity.

Figure 2: Routing Update Activity: Week starting 9th June 2015 (www.potaroo.net)

This burst of activity originated at AS4788, the network operated by Telekom Malaysia.

Telekom Malaysia is a relatively large provider in Malaysia. It's customer base has in the region of 15 million users from its own network, and it provides transit services to 64 other networks, mostly based in South East Asia. This network also appears to have 13 upstream providers. The upstream providers, and the number of prefixes seen via each provider from a vantage point located in Australia at AS131072 is as follows:

                           * this appears to be some form of routing error

Like many ISPs with a large set of upstream providers Telekom Malaysia use the technique of the announcements of more specifics to manage their traffic loads over various physical circuits. In this case Telekom Malaysia has taken the 56 IPv4 address blocks that it has been allocated from its local Regional Internet Registry and they advertise some 2,200 distinct prefixes, mostly using /24 more specific advertisements drawn from the covering aggregate address blocks. The generation of 1,144 more specifics places Telekom Malaysia as one of the larger generators of more specifics in today’s Internet, but by no means the largest. This approach is performing traffic engineering over diverse physical connections by selective announcements of more specific address prefixes may be commonplace, but it can still be complicated. And of course complicated approaches often turn into operationally fragile arrangements that are prone to mishaps.

And that’s what appeared to happen to Telekom Malaysia on 12 of June.

What the rest of the world saw was a surge of route advertisements coming out of AS4788, including some 2,500 new prefixes that had not been seen before. A couple of hours later there was a large scale withdrawal of some 5,500 prefixes, before stabilising back with the re-advertisement of some 2,500 prefixes, as shown above.

If we look at just AS4788, and look at updates coming from that AS, we can break them down into prefixes that are originated by AS4788 and prefixes where AS4788 is a transit provider. The log of origin and transit activity for AS4788 over 12 June shows a rapid rise in activity at 08:43 on that day, lasting for 9 minutes.

Figure 3: Routing Update Activity: AS4788 0830 – 0930 12 June 2015

Most of this route update activity came in the form of the announcement of routes where AS4788 was a transit AS, rather than the origination of new routes originating in AS4788.

So what happened? There are two major forms of route leak: one is where a network imports eBGP-learned routes in one part of its network and converts internal routes to eBGP-exported routes in another part of its network. The routes that are mistakenly handled in this fashion appear to other networks as if they originated in this network. The other form is where eBGP-learned routes from one peer network are mistakenly exported to another peer network. In this case the external view is that the leaked routes include this network as a “transit”.

In the case of this particular route leak we saw a transit route leak, where AS4788 was re-advertising routes learned from one routing peer to another peer.

Which routes where affected? A list of 22,577 routes is probably too much for here, but lets look at the set of neighbour ASes whose routes appeared to be re-advertised by AS4877 over this period. Here’s the top 20, as seen from a BGP vantage point located in AS4777 Japan.

This list is quite different from the list of upstream transit ASes above. The likely reason is that AS4788 is present at a number of exchange points, and is picking up a large number of routes from these exchange points. During this route incident AS4788 re-advertised these learned routes back to its upstream transits. For those routes where the transit path via AS4877 represented a shorter path, this alternate path was adopted by these transit networks and further promulgated to their routing peers.

We are probably close enough to understand what happened on 12 June inside AS4788. In the routing primer above we said that as a rule of thumb there are three routing guidelines:

• Customer-learned routes are re-distributed to customers, peers, and providers
• Peer-learned routes are re-distributed to customers but not to other peers nor to providers
• Provider-learned routes are re-distributed to customers, but not to other providers, nor to any peers.

It appears that AS4877 re-advertised peer-learned routes (probably learned from DECIX) to its upstream providers, creating a form of route hijack. This was probably due to a failure in a route policy map in an eBGP router inside AS4877.

This article has been originally published on potaroo . If you want to read more about how BGP works and how leaks like the one desribed above can be prevented, please see the orginial article for more details.