Geoff Huston

More Leaky Routes

Geoff Huston
2

Most of the time, mostly everywhere, most of the Internet appears to work just fine. Indeed, it seems to work just fine enough to the point that that when it goes wrong in a significant way then it seems to be fodder for headlines in the industry press.


 

Routy Leaks - the Register

This time the problem started in Malaysia and its major Internet service provider in the country, Telekom Malaysia, on 12 June.

 

What Happened?

Now let’s go back to 12 June and look at what happened. Certainly something happened, as can be seen in the following two figures. The first is an hour-by-hour plot of the total number of prefixes announced in the Internet’s routing table.

Figure 1: Routing Table Size: Week starting 9th June 2015 (www.potaroo.net)

It looks as if 2,500 additional routes were added to the routing system, and then a couple of hours later some 5,500 routes were withdrawn, followed by the restoration of 2,500 routes, leaving the routing system some 500 routers smaller than the original state.

A similar picture is evident when looking at the updates in BGP across the same period, where the change in the BGP table size was matched by a corresponding burst in BGP update activity.

Figure 2: Routing Update Activity: Week starting 9th June 2015 (www.potaroo.net)

This burst of activity originated at AS4788, the network operated by Telekom Malaysia.

Telekom Malaysia is a relatively large provider in Malaysia. It's customer base has in the region of 15 million users from its own network, and it provides transit services to 64 other networks, mostly based in South East Asia. This network also appears to have 13 upstream providers. The upstream providers, and the number of prefixes seen via each provider from a vantage point located in Australia at AS131072 is as follows:

AS3320 DTAG Deutsche Telekom AG, DE (7 prefixes)
AS3356 LEVEL3 - Level 3 Communications, Inc., US (1 prefix)
AS4637 ASN-TELSTRA-GLOBAL Telstra Global, HK (1,131 prefixes)
AS9304 HUTCHISON-AS-AP Hutchison Global Communications, HK (1 prefix)
AS24115 ASN-EQIX-MLPE Equinix, HK (1,296 prefixes)
AS6453 TATA COMMUNICATIONS (AMERICA) INC, US (2 prefixes)
AS2516 KDDI KDDI CORPORATION, JP (2,074 prefixes)
AS18206 VPIS-AP VADS Managed Business Internet Service Provider, MY (1 prefix *)
AS1273 CW Cable and Wireless Worldwide plc, GB (183 prefixes)
AS3257 TINET-BACKBONE Tinet SpA, DE (179 prefixes)
AS1299 TELIANET TeliaSonera AB, SE (3 prefixes)
AS6939 HURRICANE - Hurricane Electric, Inc., US (2,222 prefixes)
AS17557 PKTELECOM-AS-PK Pakistan Telecommunication Company Limited, PK (15 prefixes)


                           * this appears to be some form of routing error

Like many ISPs with a large set of upstream providers Telekom Malaysia use the technique of the announcements of more specifics to manage their traffic loads over various physical circuits. In this case Telekom Malaysia has taken the 56 IPv4 address blocks that it has been allocated from its local Regional Internet Registry and they advertise some 2,200 distinct prefixes, mostly using /24 more specific advertisements drawn from the covering aggregate address blocks. The generation of 1,144 more specifics places Telekom Malaysia as one of the larger generators of more specifics in today’s Internet, but by no means the largest. This approach is performing traffic engineering over diverse physical connections by selective announcements of more specific address prefixes may be commonplace, but it can still be complicated. And of course complicated approaches often turn into operationally fragile arrangements that are prone to mishaps.

And that’s what appeared to happen to Telekom Malaysia on 12 of June.

What the rest of the world saw was a surge of route advertisements coming out of AS4788, including some 2,500 new prefixes that had not been seen before. A couple of hours later there was a large scale withdrawal of some 5,500 prefixes, before stabilising back with the re-advertisement of some 2,500 prefixes, as shown above.

If we look at just AS4788, and look at updates coming from that AS, we can break them down into prefixes that are originated by AS4788 and prefixes where AS4788 is a transit provider. The log of origin and transit activity for AS4788 over 12 June shows a rapid rise in activity at 08:43 on that day, lasting for 9 minutes.

Figure 3: Routing Update Activity: AS4788 0830 – 0930 12 June 2015

Most of this route update activity came in the form of the announcement of routes where AS4788 was a transit AS, rather than the origination of new routes originating in AS4788.

So what happened? There are two major forms of route leak: one is where a network imports eBGP-learned routes in one part of its network and converts internal routes to eBGP-exported routes in another part of its network. The routes that are mistakenly handled in this fashion appear to other networks as if they originated in this network. The other form is where eBGP-learned routes from one peer network are mistakenly exported to another peer network. In this case the external view is that the leaked routes include this network as a “transit”.

In the case of this particular route leak we saw a transit route leak, where AS4788 was re-advertising routes learned from one routing peer to another peer.

Which routes where affected? A list of 22,577 routes is probably too much for here, but lets look at the set of neighbour ASes whose routes appeared to be re-advertised by AS4877 over this period. Here’s the top 20, as seen from a BGP vantage point located in AS4777 Japan.

Prefixes ASN Description
5,545 6695 DECIX-AS DE-CIX Management GmbH, DE
3,699 9394 CTTNET China TieTong Telecommunications Corporation, CN
3,381 1273 CW Cable and Wireless Worldwide plc, GB
1,594 4788 TMNET-AS-AP TM Net, Internet Service Provider, MY (Announced Prefixes)
818 8359 MTS MTS OJSC, RU
781 38285 M2TELECOMMUNICATIONS-AU M2 Telecommunications Group Ltd, AU
643 2119 TELENOR-NEXTEL Telenor Norge AS, NO
561 3209 VODANET Vodafone GmbH, DE
422 8708 RCS-RDS RCS & RDS SA, RO
365 17557 PKTELECOM-AS-PK Pakistan Telecommunication Company Limited, PK
260 16150 PORT80-GLOBALTRANSIT Availo Networks AB, SE
242 12552 IPO-EU IP-Only Networks AB, SE
241 23520 COLUMBUS-NETWORKS - Columbus Networks USA, Inc., US
220 3741 IS, ZA
186 4635 HKIX-RS1 Hong Kong Internet Exchange--Route Server 1, HK
162 41095 IPTP IPTP LTD, NL
153 5588 GTSCE T-Mobile Czech Republic a.s., CZ
147 2647 SITA Societe Internationale de Telecommunications Aeronautiques, FR
144 8426 CLARANET-AS ClaraNET LTD, GB
143 2686 ATGS-MMD-AS - AT&T Global Network Services, LLC, US

 

This list is quite different from the list of upstream transit ASes above. The likely reason is that AS4788 is present at a number of exchange points, and is picking up a large number of routes from these exchange points. During this route incident AS4788 re-advertised these learned routes back to its upstream transits. For those routes where the transit path via AS4877 represented a shorter path, this alternate path was adopted by these transit networks and further promulgated to their routing peers.

We are probably close enough to understand what happened on 12 June inside AS4788. In the routing primer above we said that as a rule of thumb there are three routing guidelines:

  • Customer-learned routes are re-distributed to customers, peers, and providers
  • Peer-learned routes are re-distributed to customers but not to other peers nor to providers
  • Provider-learned routes are re-distributed to customers, but not to other providers, nor to any peers.

It appears that AS4877 re-advertised peer-learned routes (probably learned from DECIX) to its upstream providers, creating a form of route hijack. This was probably due to a failure in a route policy map in an eBGP router inside AS4877.

 

This article has been originally published on potaroo . If you want to read more about how BGP works and how leaks like the one desribed above can be prevented, please see the orginial article for more details.

2

You may also like

View more

About the author

Geoff Huston is the Chief Scientist at APNIC, where he undertakes research on topics associated with Internet infrastructure, IP technologies, and address distribution policies. From 1995 to 2005, Geoff was the Chief Internet Scientist at Telstra, where he provided a leading role in the construction and further development of Telstra's Internet service offerings, both in Australia and as part of Telstra's global operations. Prior to Telstra, Mr Huston worked for the Australian National University, where he led the initial construction of the Internet in Australia in the late 1980s as the Technical Manager of the Australian Academic and Research Network. He has authored a number of books dealing with IP technology, as well as numerous papers and columns. He was a member of the Internet Architecture Board from 1999 until 2005 and served as its Executive Director from 2001 to 2005. He is an active member of the Internet Engineering Task Force, where he currently chairs two Working Groups. He served on the Board of Trustees of the Internet Society from 1992 until 2001 and served a term as Chair of the Board in 1999. He has served on the Board of the Public Internet Registry and also on the Executive Council of APNIC. He chaired the Internet Engineering and Planning Group from 1992 until 2005.

Comments 2