You are here: Home > Publications > RIPE Labs > Geoff Huston > More Leaky Routes

More Leaky Routes

Geoff Huston — 03 Jul 2015
Most of the time, mostly everywhere, most of the Internet appears to work just fine. Indeed, it seems to work just fine enough to the point that that when it goes wrong in a significant way then it seems to be fodder for headlines in the industry press.

 

Routy Leaks - the Register

This time the problem started in Malaysia and its major Internet service provider in the country, Telekom Malaysia, on 12 June.

 

What Happened?

Now let’s go back to 12 June and look at what happened. Certainly something happened, as can be seen in the following two figures. The first is an hour-by-hour plot of the total number of prefixes announced in the Internet’s routing table.

Figure 1: Routing Table Size: Week starting 9th June 2015 (www.potaroo.net)

It looks as if 2,500 additional routes were added to the routing system, and then a couple of hours later some 5,500 routes were withdrawn, followed by the restoration of 2,500 routes, leaving the routing system some 500 routers smaller than the original state.

A similar picture is evident when looking at the updates in BGP across the same period, where the change in the BGP table size was matched by a corresponding burst in BGP update activity.

Figure 2: Routing Update Activity: Week starting 9th June 2015 (www.potaroo.net)

This burst of activity originated at AS4788, the network operated by Telekom Malaysia.

Telekom Malaysia is a relatively large provider in Malaysia. It's customer base has in the region of 15 million users from its own network, and it provides transit services to 64 other networks, mostly based in South East Asia. This network also appears to have 13 upstream providers. The upstream providers, and the number of prefixes seen via each provider from a vantage point located in Australia at AS131072 is as follows:

AS3320  DTAG Deutsche Telekom AG, DE (7 prefixes)
AS3356  LEVEL3 - Level 3 Communications, Inc., US (1 prefix)
AS4637  ASN-TELSTRA-GLOBAL Telstra Global, HK (1,131 prefixes)
AS9304  HUTCHISON-AS-AP Hutchison Global Communications, HK (1 prefix)
AS24115  ASN-EQIX-MLPE Equinix, HK (1,296 prefixes)
AS6453  TATA COMMUNICATIONS (AMERICA) INC, US (2 prefixes)
AS2516  KDDI KDDI CORPORATION, JP (2,074 prefixes)
AS18206  VPIS-AP VADS Managed Business Internet Service Provider, MY (1 prefix *)
AS1273  CW Cable and Wireless Worldwide plc, GB (183 prefixes)
AS3257  TINET-BACKBONE Tinet SpA, DE (179 prefixes)
AS1299  TELIANET TeliaSonera AB, SE (3 prefixes)
AS6939  HURRICANE - Hurricane Electric, Inc., US (2,222 prefixes)
AS17557 PKTELECOM-AS-PK Pakistan Telecommunication Company Limited, PK (15 prefixes)


                           * this appears to be some form of routing error

Like many ISPs with a large set of upstream providers Telekom Malaysia use the technique of the announcements of more specifics to manage their traffic loads over various physical circuits. In this case Telekom Malaysia has taken the 56 IPv4 address blocks that it has been allocated from its local Regional Internet Registry and they advertise some 2,200 distinct prefixes, mostly using /24 more specific advertisements drawn from the covering aggregate address blocks. The generation of 1,144 more specifics places Telekom Malaysia as one of the larger generators of more specifics in today’s Internet, but by no means the largest. This approach is performing traffic engineering over diverse physical connections by selective announcements of more specific address prefixes may be commonplace, but it can still be complicated. And of course complicated approaches often turn into operationally fragile arrangements that are prone to mishaps.

And that’s what appeared to happen to Telekom Malaysia on 12 of June.

What the rest of the world saw was a surge of route advertisements coming out of AS4788, including some 2,500 new prefixes that had not been seen before. A couple of hours later there was a large scale withdrawal of some 5,500 prefixes, before stabilising back with the re-advertisement of some 2,500 prefixes, as shown above.

If we look at just AS4788, and look at updates coming from that AS, we can break them down into prefixes that are originated by AS4788 and prefixes where AS4788 is a transit provider. The log of origin and transit activity for AS4788 over 12 June shows a rapid rise in activity at 08:43 on that day, lasting for 9 minutes.

Figure 3: Routing Update Activity: AS4788 0830 – 0930 12 June 2015

Most of this route update activity came in the form of the announcement of routes where AS4788 was a transit AS, rather than the origination of new routes originating in AS4788.

So what happened? There are two major forms of route leak: one is where a network imports eBGP-learned routes in one part of its network and converts internal routes to eBGP-exported routes in another part of its network. The routes that are mistakenly handled in this fashion appear to other networks as if they originated in this network. The other form is where eBGP-learned routes from one peer network are mistakenly exported to another peer network. In this case the external view is that the leaked routes include this network as a “transit”.

In the case of this particular route leak we saw a transit route leak, where AS4788 was re-advertising routes learned from one routing peer to another peer.

Which routes where affected? A list of 22,577 routes is probably too much for here, but lets look at the set of neighbour ASes whose routes appeared to be re-advertised by AS4877 over this period. Here’s the top 20, as seen from a BGP vantage point located in AS4777 Japan.

Prefixes ASN Description
5,545 6695  DECIX-AS DE-CIX Management GmbH, DE
3,699 9394  CTTNET China TieTong Telecommunications Corporation, CN
3,381 1273  CW Cable and Wireless Worldwide plc, GB
1,594 4788  TMNET-AS-AP TM Net, Internet Service Provider, MY (Announced Prefixes)
818 8359  MTS MTS OJSC, RU
781 38285  M2TELECOMMUNICATIONS-AU M2 Telecommunications Group Ltd, AU
643 2119  TELENOR-NEXTEL Telenor Norge AS, NO
561 3209  VODANET Vodafone GmbH, DE
422 8708  RCS-RDS RCS & RDS SA, RO
365 17557  PKTELECOM-AS-PK Pakistan Telecommunication Company Limited, PK
260 16150  PORT80-GLOBALTRANSIT Availo Networks AB, SE
242 12552  IPO-EU IP-Only Networks AB, SE
241 23520  COLUMBUS-NETWORKS - Columbus Networks USA, Inc., US
220 3741  IS, ZA
186 4635  HKIX-RS1 Hong Kong Internet Exchange--Route Server 1, HK
162 41095  IPTP IPTP LTD, NL
153 5588  GTSCE T-Mobile Czech Republic a.s., CZ
147 2647  SITA Societe Internationale de Telecommunications Aeronautiques, FR
144 8426  CLARANET-AS ClaraNET LTD, GB
143 2686  ATGS-MMD-AS - AT&T Global Network Services, LLC, US

 

This list is quite different from the list of upstream transit ASes above. The likely reason is that AS4788 is present at a number of exchange points, and is picking up a large number of routes from these exchange points. During this route incident AS4788 re-advertised these learned routes back to its upstream transits. For those routes where the transit path via AS4877 represented a shorter path, this alternate path was adopted by these transit networks and further promulgated to their routing peers.

We are probably close enough to understand what happened on 12 June inside AS4788. In the routing primer above we said that as a rule of thumb there are three routing guidelines:

  • Customer-learned routes are re-distributed to customers, peers, and providers
  • Peer-learned routes are re-distributed to customers but not to other peers nor to providers
  • Provider-learned routes are re-distributed to customers, but not to other providers, nor to any peers.

It appears that AS4877 re-advertised peer-learned routes (probably learned from DECIX) to its upstream providers, creating a form of route hijack. This was probably due to a failure in a route policy map in an eBGP router inside AS4877.

 

This article has been originally published on potaroo . If you want to read more about how BGP works and how leaks like the one desribed above can be prevented, please see the orginial article for more details.

2 Comments

Thomas King says:
04 Jul, 2015 04:02 PM
Great read! Thanks for the detailed analysis. However, I assume Figure 1 and 2 are from 2015. :-)
Geoff Huston says:
05 Jul, 2015 02:08 PM
Heh - It wouldn't be me if I didn't include a few typos. Thanks for spotting this and letting me know, and thanks to the magic blogmaster elves for fixing this!
Add comment

You can add a comment by filling out the form below. Only plain text is possible. Web and email addresses will be transformed into clickable links. Comments are moderated so they won't appear immediately.