NTT’s RPKI Deployment - Monitoring, Awareness, and Community

Over the last two and a half years, across various events, Massimo has been encouraging the network operator community to deploy RPKI, particularly Route Origin Validation (ROV), in networks managed by NTT to help secure the Internet’s routing infrastructure. The following is a summary of what Massimo has shared about NTT’s RPKI deployment.

A multi-year project

Tier 1 provider NTT (Figure 1) operates one of the largest backbone networks (AS2914) in the world. In March 2020, NTT announced it had deployed RPKI-based Border Gateway Protocol (BGP) Origin Validation.

ROV is considered the second of the basic two steps required to deploy RPKI. The first step is to create Route Origin Authorisations (ROAs), cryptographically signed documents, stored in a public repository, that declare which Autonomous System (AS) is authorised to announce a certain prefix in BGP routing. ROV is performed by processing ROAs into a set of validated routes and using this to make BGP routing decisions.

Those of you reading in the RIPE NCC service region can go here to learn more about creating ROAs.

A router implementing ROV rejects any routes that conflict with a signed ROA (but may still permit routes that are unsigned to pass) — a daunting proposition for any operator let alone one that is among the top five largest networks in the world. Regardless, NTT considers RPKI a critical commitment to reducing the negative impact of misconfigurations or malicious attacks in the global Internet routing system.

Given the importance of a successful deployment, NTT’s RPKI deployment was years in the making, involving extensive outreach, education, and collaboration with customers, industry partners and internal teams. But, as Massimo has reported, it’s not something that NTT has been able to set and forget. Rather, it requires continual monitoring and awareness.  

“Unfortunately, I see a lot of workshops and tutorials that focus on signing ROAs and then RPKI is done. The reality is that the RPKI is a new layer that you will have to keep in mind every time you do your daily operations,” Massimo said during RIPE 84.

The importance of monitoring

For Massimo, RPKI monitoring is a “…fundamental activity to mitigate or to timely correct invalid announcements [that] would impact the reachability of your services”.

As part of NTT’s routing security efforts, Massimo developed BGPalerter — an open-source monitoring app now used by hundreds of organisations to monitor BGP and RPKI. As discussed in the one-year review of RPKI operations at NTT, BGPalerter alerts you (via email or messaging app) if:

• Any prefixes lose visibility
• Any prefixes are hijacked
• There are unexpected peer ASes appearing in AS paths
• Your AS is announcing RPKI invalid prefixes
• ROAs are expiring
• There is an ongoing Trust Anchor malfunction
• A ROA involving any of your prefixes or ASes is added/edited/deleted

Massimo has given several presentations on this tool, including at NLNOG, INNOG, and RIPE meetings, and wrote a useful tutorial on how to use BGPalerter to monitor BGP prefixes.

Recently, he introduced PacketVis a service based on BGPalerter for those that do not want to run the app themselves.

In a presentation at RIPE 84, Massimo discussed the following common causes for RPKI invalid announcements that he and his team had found to date, and subsequent actions taken by NTT’s Network Operations Centre (NOC) to resolve these alerts:

• NTT announced a new prefix, but due to the existence of a ROA covering a less-specific prefix it resulted in a violation of the max length (~57% of all alerts)
• NTT migrated prefixes from one AS to another (change of origin), but it didn’t properly update its ROAs (~25%)
• NTT announced a customer’s prefix (not RPKI unknown), but the customer did not have a ROA for NTT’s AS (~17%)

From this baseline understanding, the NTT team developed more advanced RPKI monitoring features for BGPalerter, an automation system, and new strict procedures that helped to reduce RPKI invalid announcements by around 87%.

Throughout his updates, Massimo has often cited a community mindset and the importance of open-sourcing tools as crucial to successful RPKI deployment. Leading by example, NTT developed BGPalerter for their own network and then open-sourced the software to benefit the network operator community at large and encourage RPKI deployment to help secure the Internet’s routing infrastructure.

He also cites the routing collection project - RIPE RIS - and the concrete actions of MANRS as helpful and community-minded initiatives for routing security. There’s no doubt NTT’s deployment has been successful based on these measures.

The percentage of routable address space in BGP covered by a ROA is a good measure of deployment. As shown in Figure 3, NTT’s AS2914 is currently above 80%. RPKI-valid prefixes are a good measure of successful deployment. Currently, BGP Toolkit reports AS2914 as routing through 14 Internet Exchanges, originating 267 IPv4 and 10 IPv6 prefixes for a total of 277. Of those, 227 prefixes matched in ROV are valid, and none are invalid.

As a MANRS Ambassador, Massimo supports the community-driven best practices and network operator actions of MANRS, and as perfectly summarised in his most recent update at RIPE 85: You run out of excuses! It’s time to monitor your BGP and RPKI operations.

Well said, Massimo.

You can get lots more information on RPKI - including documentation on deploying RPKI and managing ROAs - from both the RIPE NCC website and over at the APNIC portal.

Sources for figures:

• Figure 1 - About NTT and the global IP network
• Figure 2 - Massimo's presentation at RIPE 84
• Figure 3 - APNIC Labs

This article was originally published over on the APNIC blog.