Unknown Attribute 28 - A Source Of Entropy in Interdomain Routing?

Multiple sources recently notified us of an event that occurred in the Interdomain routing system on Friday 2 June. Initial reports suggested that something was having a noticeable effect on interdomain routing, specifically in IPv6.

Having started to look more closely into what exactly was going on, my interpretation so far is that packets with a deprecated BGP attribute - attribute number 28, defined in RFC6790, also known as "BGP Entropy Label Capability Attribute" - caused session resets on particular versions of Juniper routers, due to them incorrectly flagging these packets as corrupt.

Unknown attributes

If you know the nitty gritty of BGP, you probably get the significance of this, but it's kinda hard to get people outside of this field up to speed.

For present purposes, all we need say about BGP attributes is that they’re specific bits of information - contained in BGP messages - that help routers determine which routes are best for getting packets to their destinations. Some of these BGP attributes are mandatory (must be recognised by all routers), but others are entirely optional - and that’s important, because it means routers can and do vary as to which attributes they’re actually set up to recognise.

When a router encounters an attribute it doesn’t recognise, it’ll typically ignore it. But what appears to have happened in this case is that the presence of an unknown attribute caused some routers to read incoming BGP messages as malformed.

One peculiarity of old-school BGP is that if it encounters malformed BGP packets, the BGP session is reset. But this extreme response is pretty undesirable, and so there’s an RFC defining better behaviour (RFC7606, August 2015). From that RFC:

This behavior is undesirable because a session reset impacts not only routes with the offending attribute but also other valid routes exchanged over the session. In the case of optional transitive attributes, the behavior is especially troublesome and may present a potential security vulnerability. This is because attributes may have been propagated without being checked by intermediate routers that don't recognize the attributes. In effect, the attributes may have been tunneled; when they reach a router that recognizes and checks the attributes, the session that is reset may not be associated with the router that is at fault. To make matters worse, in such cases, although the problematic attributes may have originated with a single update transmitted by a single BGP speaker, by the time they encounter a router that checks them they may have been replicated many times and thus may cause the reset of many peering sessions. Thus, the damage inflicted may be multiplied manyfold.

The event of Friday 2 June looks like a case of this undesirable behaviour, so we look into RIS what we can find and learn about this.

What we found so far

We analysed all updates that RIS collected on June 2 2023 (all times in UTC). We indeed find a total of 417 BGP update messages that have this unknown attribute. The first messages we found where at 13:33:46 and look like this:

The last message of this type we saw at 16:34:12 .

These messages were seen at 82 of our RIS route collector peering sessions (out of 1527 total now, this total includes "full" and partial feeds), for 66 peer ASNs (out of 609 total now), across 18 different route collectors (out of 23 total now), which gives an idea of the spread of the event.

These 2 initial messages you see above contain the two IPv6 prefixes we saw in all updates containing BGP attribute 28. In some of the BGP messages (that can contain multiple prefixes), we saw other prefixes too. This led to a specific distribution of prefixes seen in the BGP attribute 28-containing BGP messages.¹ The AS paths we see in these messages are also interesting.² (A full list of these both prefixes and AS Paths is available in the notes section below, or you can also check out the Github repo for a file with the complete list of BGP messages.)

The AS paths all have the network AS262494 (Virtex, BR) in them, the origin AS being either AS269480 or AS264366. We see AS6939 (Hurricane Electric) as upstream in most AS paths, which is not surprising given Hurricane Electric's prominence in IPv6 transit in general. Apart from paths via AS6939, we also see paths via RIS direct peers AS41047, AS51185, and AS61595.

Wider effects

This figure shows the effects this event had on a particular network, as seen in the Internet Health Report (IHR) AS-Hegemony:

The top graph shows fluctuations in the dependance on upstreams of this network, while the lower graph shows how for the timeframe the BGP event happened this network lost some networks that depended on them. This graph also raises more questions: AS-Hegemony scores are calculated for IPv4, and the event, at least as far as we see in RIS, was IPv6. How did that happen?

As we don’t want to emphasise the effects for a single network, but at the same time want to show what an impact this can have, we chose not to name the particular network.

Another network (a large CDN) told us "this was very visible on our end", so there might be other evidence of this event in the routing data that we collect with RIS, or in other route collection systems.

We looked for traffic graphs (IXPs and elsewhere) that give an idea of the amount of traffic that got diverted because of this, but we didn't see any unusual patterns correlated with what we see in RIS due to this event. This could suggest that, despite the reported effects at some networks, this event went by without noticeable effect on the rest of the Internet. This could have been helped due to mitigation efforts the affected networks took.

Why is this event interesting?

This event might have been relatively small in scope, but it raises quite a lot of questions for me. For example, while this event was limited to IPv6, it is worth asking what would happen if a similar event were to take place in IPv4. Indeed, it’s worth considering in more detail whether this could happen in IPv4 and just what the fallout would be. Or what would happen in a strictly single-stack world?

Another thing to consider is how to secure networks against this type of event. For networks with Juniper gear for instance, there is in fact a particular document that describes mitigation against attribute-related issues.

So those who use this gear are recommended to consider changing configs according to this. Those using other gear are encouraged to find if/how these platforms mitigate against this type of thing.

It’s worth nothing that we’ve looked into BGP attributes that were unknown to the bgpdump parser before here on RIPE Labs and from that we distilled a list of unknown attributes that were seen in the wild at that point. At that time we also found packets with BGP attribute 28. It may be an interesting research project to look how the use and visibility of these attributes changed over time.

Further analysis and open questions

It is not easy parsing BGP/MRT data with these types of attributes. For this case, the only thing that worked for me was using bgpdump in multi-line format, and do custom post-processing, which isn’t a particularly speedy way to get this signal out of the data we collect.

As an immediate consequence of this event, I know that the BGPKIT suite of MRT processing tools made improvements to specifically handle unknown and deprecated BGP attributes better. The next time I want to analyse this type of event I'll certainly consider using BGPKIT for this.

Open Questions

Here are some of the questions this raises with me:

• To what extent is the complexity of BGP becoming a problem for the Internet? This is not a new question. Just as the DNS has this issue - it even has a name and animal attached to it: the DNS Camel! - it could be that BGP has hit a level of complexity that hampers the evolution and operation of Internet interdomain routing. Protocol-camel started of with DNS, but now also lists BGP and NTP.

• As our selection of route collector peers is quite biased (“clue core”) what was the true extent of this event, and to what extent are we hampered in our view of the Internet by the places where we don’t collect BGP data for?

• To what extent are BGP implementations tested and hardened? I consider Juniper one of the high-end vendors in this field, and I think they spend considerable effort on quality assurance and testing. I’m actually more worried about vendors who have less resources to spend on quality assurance and testing. What bugs are lurking here? To what extent are BGP implementations fuzzed? A quick Internet search revealed that at least some of this is happening, see for instance this recent item about vulnerabilities found by BGP fuzzing.

Lots of things to think about. I hope people have some answers. It would be great to engage more on this in our thread over on the RIPE NCC forum.

Notes

1. Distribution of prefixes seen in the BGP attribute 28-containing BGP messages

216 2804:1d64::/32

201 2804:6548::/32

1 2c0f:f800::/28

1 2c0f:ef18:8000::/36

1 2a0f:7600::/32

1 2a0b:1306:3::/48

1 2a0a:d6c0:208::/45

1 2a05:8440::/29

1 2a04:fcc0::/29

1 2a04:c007::/32

1 2a02:2bc0::/32

1 2a01:c0::/32

1 2a01:a000::/32

1 2a01:50a0::/32

1 2a00:d680::/32

1 2a00:c00:f062::/47

1 2a00:c00:f060::/47

1 2a00:c00:f060::/46

1 2a00:6ec0:400::/40

1 2a00:6ec0:300::/40

1 2a00:6ec0:100::/40

1 2a00:6480:8748::/48

1 2606:ecc0:2001::/48

1 2001:67c:484::/48

1 2001:67c:1808::/48

2. AS paths

18 ASPATH: 6939 262494 264366

17 ASPATH: 6939 262494 269480

17 ASPATH: 15547 6939 262494 269480

17 ASPATH: 15547 6939 262494 264366

9 ASPATH: 211380 41051 6939 262494 264366

7 ASPATH: 924 6939 262494 264366

7 ASPATH: 58057 6939 262494 269480

7 ASPATH: 205593 6939 262494 269480

6 ASPATH: 924 6939 262494 269480

6 ASPATH: 58057 6939 262494 264366

6 ASPATH: 211380 41051 6939 262494 269480

6 ASPATH: 205593 6939 262494 264366

6 ASPATH: 199524 6939 262494 264366

5 ASPATH: 51519 6939 262494 264366

5 ASPATH: 204708 6939 262494 264366

5 ASPATH: 202032 6939 262494 264366

5 ASPATH: 199524 6939 262494 269480

4 ASPATH: 835 6939 262494 269480

4 ASPATH: 835 6939 262494 264366

4 ASPATH: 51519 6939 262494 269480

4 ASPATH: 41047 262494 264366

4 ASPATH: 34927 6939 262494 269480

4 ASPATH: 34927 6939 262494 264366

4 ASPATH: 28824 50304 6939 262494 264366

4 ASPATH: 204708 6939 262494 269480

4 ASPATH: 202032 6939 262494 269480

3 ASPATH: 61138 6939 262494 264366

3 ASPATH: 58308 6939 262494 264366

3 ASPATH: 57695 6939 262494 269480

3 ASPATH: 57695 6939 262494 264366

3 ASPATH: 57381 6939 262494 264366

3 ASPATH: 51185 262494 269480

3 ASPATH: 51185 262494 264366

3 ASPATH: 50304 6939 262494 264366

3 ASPATH: 41157 6939 262494 264366

3 ASPATH: 41047 262494 269480

3 ASPATH: 37239 6939 262494 269480

3 ASPATH: 34927 34927 6939 262494 264366

3 ASPATH: 34177 6939 262494 264366

3 ASPATH: 328137 6939 262494 269480

3 ASPATH: 327960 6939 262494 269480

3 ASPATH: 327960 6939 262494 264366

3 ASPATH: 29504 6939 262494 269480

3 ASPATH: 29504 6939 262494 264366

3 ASPATH: 213045 6939 262494 269480

3 ASPATH: 209718 209022 6939 262494 264366

3 ASPATH: 207841 6939 262494 269480

3 ASPATH: 207841 6939 262494 264366

3 ASPATH: 207564 6939 262494 264366

3 ASPATH: 16347 6939 262494 264366

2 ASPATH: 924 12186 6939 262494 264366

2 ASPATH: 8888 6939 262494 269480

2 ASPATH: 8607 6939 262494 269480

2 ASPATH: 8607 6939 262494 264366

2 ASPATH: 6720 1853 6939 262494 269480

2 ASPATH: 6720 1853 6939 262494 264366

2 ASPATH: 6233 6939 262494 269480

2 ASPATH: 6233 6939 262494 264366

2 ASPATH: 61218 6939 262494 269480

2 ASPATH: 61218 6939 262494 264366

2 ASPATH: 61138 6939 262494 269480

2 ASPATH: 60557 6939 262494 269480

2 ASPATH: 60557 6939 262494 264366

2 ASPATH: 59891 6939 262494 269480

2 ASPATH: 58308 6939 262494 269480

2 ASPATH: 57821 6939 262494 269480

2 ASPATH: 57821 6939 262494 264366

2 ASPATH: 57381 6939 262494 269480

2 ASPATH: 56910 6939 262494 269480

2 ASPATH: 56910 6939 262494 264366

2 ASPATH: 56430 6939 262494 269480

2 ASPATH: 56430 6939 262494 264366

2 ASPATH: 51786 6939 262494 269480

2 ASPATH: 51786 6939 262494 264366

2 ASPATH: 50304 6939 262494 269480

2 ASPATH: 50058 6939 262494 269480

2 ASPATH: 50058 6939 262494 264366

2 ASPATH: 48821 6939 262494 269480

2 ASPATH: 48821 6939 262494 264366

2 ASPATH: 48292 6939 262494 269480

2 ASPATH: 48292 6939 262494 264366

2 ASPATH: 46997 6939 262494 269480

2 ASPATH: 46997 6939 262494 264366

2 ASPATH: 44103 6939 262494 269480

2 ASPATH: 44103 6939 262494 264366

2 ASPATH: 41157 6939 262494 269480

2 ASPATH: 394414 6939 262494 269480

2 ASPATH: 394414 6939 262494 264366

2 ASPATH: 37989 6939 262494 264366

2 ASPATH: 37239 6939 262494 264366

2 ASPATH: 34927 34927 6939 262494 269480

2 ASPATH: 34177 6939 262494 269480

2 ASPATH: 3212 6939 262494 269480

2 ASPATH: 3212 6939 262494 264366

2 ASPATH: 28824 50304 6939 262494 269480

2 ASPATH: 21700 6939 262494 269480

2 ASPATH: 21700 6939 262494 264366

2 ASPATH: 21412 6939 262494 269480

2 ASPATH: 21412 6939 262494 264366

2 ASPATH: 213045 6939 262494 264366

2 ASPATH: 212123 6939 262494 269480

2 ASPATH: 212123 6939 262494 264366

2 ASPATH: 210633 6939 262494 269480

2 ASPATH: 210633 6939 262494 264366

2 ASPATH: 209718 209022 6939 262494 269480

2 ASPATH: 207564 6939 262494 269480

2 ASPATH: 206499 6939 262494 269480

2 ASPATH: 206499 6939 262494 264366

2 ASPATH: 206271 6939 262494 269480

2 ASPATH: 206271 6939 262494 264366

2 ASPATH: 204508 6939 262494 269480

2 ASPATH: 204508 6939 262494 264366

2 ASPATH: 198249 6939 262494 269480

2 ASPATH: 198249 6939 262494 264366

2 ASPATH: 16347 6939 262494 269480

2 ASPATH: 15435 6939 262494 269480

2 ASPATH: 15435 6939 262494 264366

2 ASPATH: 13830 6939 262494 269480

2 ASPATH: 13830 6939 262494 264366

2 ASPATH: 12969 6939 262494 269480

2 ASPATH: 12969 6939 262494 264366

2 ASPATH: 12307 6939 262494 269480

2 ASPATH: 12307 6939 262494 264366

1 ASPATH: 924 12186 6939 262494 269480

1 ASPATH: 6720 8447 6939 262494 269480

1 ASPATH: 61595 262494 269480

1 ASPATH: 61595 262494 264366

1 ASPATH: 59891 6939 262494 264366

1 ASPATH: 394414 62943 6939 262494 269480

1 ASPATH: 394414 62943 6939 262494 264366

1 ASPATH: 37989 6939 262494 269480

1 ASPATH: 35426 6939 262494 264366

1 ASPATH: 328137 6939 262494 264366

1 ASPATH: 212934 6939 262494 269480

1 ASPATH: 212934 6939 262494 264366

1 ASPATH: 142289 50058 6939 262494 269480

1 ASPATH: 142289 50058 6939 262494 264366