Researching Next Generation RIS Route Collectors

Note: The full research report can be found in Appendix B .

Please also see the second part of this project: Building the Next Generation RIS Collectors.

Introduction

The Routing Information Service (RIS) , provided by the RIPE NCC, was established in 2001 and started with the idea to collect and store Internet routing data from several locations around the globe. This data, which is collected and saved by the RIS Remote Route Collectors, can be accessed via RIPEstat and used for further analyses. A Remote Route Collector (RRC), as the name suggests, collects all the Internet routing data from several points around the world and stores this data for further processing. This data is saved in the MRT Routing Information Export format and used by different applications, both by the community and internal RIPE NCC systems.

The current implementation runs in a batch-oriented mode that involves a lot of delay. Even though there are new requirements, the system still relies on the data collected by the RRCs. Therefore we need to look at possible alternatives that could be used to collect and store the Internet routing data. Before it is even possible to look at the possible alternatives that exist, we need to first look at the current requirements.

Requirements

During the first few weeks of the internship, I found out which requirements currently exist. New feature requests from the community regarding the RRCs have been added to the requirements list.

Now that the requirements are known, it would be useful to prioritise them. We can give each requirement a priority using the MoSCoW method . After talking to my colleagues at the RIPE NCC, we agreed on the final MoSCoW scheme, which was used throughout the entire project. You can find the complete list of requirements (based on the MoSCow scheme) in Appendix A .

One of the most important requirements is the requirement to maintain the Multi-threaded Routing Toolkit (MRT) formatted files for backwards compatibility. These files are currently used by the community and internal systems of the RIPE NCC to process the data. The MRT Routing Information Export Format is the de facto standard for storing Internet routing data. Since all processes rely on these files, the next generation of RIS should also facilitate this.

There is one other requirement that is very important, and that is also one of the reasons why this project has been initiated. It is the requirement for new RRCs to allow for lower delays between the data collection and the visibility in the datasets. The goal of the new RRCs is to allow future implementation to gain less delay with the overall RIS system. It involves changes that have to be made in the overall processing pipeline.

Potential Solutions

An application or project was considered a 'potential solution' when it could meet at least one of the requirements. The software solutions covered below are all considered to be within the scope of this research project.

ExaBGP

ExaBGP , the "BGP swiss army knife of networking" (and I now definitely concur with that), was considered a potential solution. Since we would like to move from the batch-oriented process to something more stream-oriented or event-driven, ExaBGP could be used as the core of the whole prototype and provide other applications with the data that it receives. Unfortunately it does not maintain the RIB state, which could also be one of ExaBGP's strengths since it provides you with the information that you want and any further processing depends on your application. This gives you the freedom to use ExaBGP in many use cases.

Since it does not maintain a RIB state, another application should provide this feature. But which application? The goal is to develop a modular, extensible solution that does not only support the current requirements, but also allows for future changes. The illustration below describes this philosophy.

Figure 1: Data flow and design philosophy of the prototype

BGPmon

BGPmon meets our requirements, with one limitation: It is not possible to announce routes with BGPmon , which implies that we cannot meet one of the "must-have" requirements. It is a design decision of the BGPmon team, so it will quite likely never be supported by BGPmon. This puts a serious limitation on the use of BGPmon. We could modify BGPmon to add support for announcements, but even then the changes will never make it into the mainstream distribution, which means we would have to maintain it.

OSR Quagga

This is a Quagga version maintained by the OSR group and it pushes their modifications to the mainstream Quagga release. Their Github page also explicitly tells visitors to use the regular Quagga releases. Ignoring this warning and cloning their repository, I could not find any changes that would make this version more usable than the mainstream Quagga releases.

BIRD

BIRD performs faster than Quagga, as was outlined in a presentation at NANOG 48 following presentation . If it is decided to use BIRD, we would basically end up with the same approach as we are using now. It could be faster and maybe allows us to connect with more peers, but when you look at the requirements, you see that we want more than only having better performance.

OpenBGPD

OpenBGPD also performs faster than Quagga, as you can see in the presentation referred to above. However, as with BIRD, we would still end up with the same implementation.

XORP

The eXtensible Open Router Platform sounds like a solution which could provide us with the features we want. It is stated that XORP is designed to be secure and, even in the event of a process failure, other processes should not be affected. If we want to create a real extensible and modular application that relies on the data of XORP, more time would be required to develop the applications than with the other alternatives.

Vyatta

If you want to have a firewall, router, VPN, and traffic management system Vyatta could be an interesting solution. However, looking at the requirements that we have, we want to have access to the BGP messages. Moreover, Quagga is used at the core for routing, so it would again be a same approach that we are using right now, since Quagga would create the MRT-formatted files.

PyRT

Python Routing Toolkit is fairly old, so old that their website certificate has expired (it also has an invalid root certificate). It is only a BGP listener and does not send any messages. You might ask, is it the same problem as with BGPmon? Yes, but it also does not send any keepalive messages, which results in peers terminating the connections after some time.

BMP

The BGP monitoring protocol is a protocol that really suits our use cases. The only disadvantages right now are that it is still a draft and that all neighbours have to support BMP. We want to be able to connect to as many neighbours as possible and this will be a real restriction in choosing our peering neighbours. The other downside is, since the protocol is relatively new and still a draft, not many applications support this protocol. It could be the case that we have to develop a server application for storing and processing the messages. The concept is really good, all the messages stored in the Adj-RIB-In of the neighbour would be send to the monitoring stations which will send the messages to the server for further processing. The messages are encapsulated in Route Monitoring messages and must be parsed by another application (which must be developed to meet our use cases).

Ryu

Ryu is a component-based software-defined networking framework. It is a framework that could be used to receive BGP updates and to do further processing upon this data. In order to support the use-cases that we have, an application needs to be developed that uses the Ryu framework. Ryu itself uses multiple processes and threads to provide all the functionalities to the application that is using this framework. It is possible to use the framework to encode and decode the BGP packets. The only downside is that an application needs to be developed that maintains the BGP connections and the state. It would involve a lot more work then the applications described previously, since they provide the same functions and already maintain the BGP connection.

Making a Decision

Now that we know what all the potential solutions are capable of, a cost/benefits scheme was created. In our case, the cost was the time that needed to be invested in extending the feature set or modifying some parts of the potential solution. A relatively high score means that less to no work is involved in meeting all the requirements, whereas a relatively low score means that a lot of work is involved in meeting the requirements. We also examiend whether it was realistic to choose a certain 'potential solution' and in the attached research paper you can read how we have made the decision.

Conclusion

ExaBGP came out on top if you look at the total score of the cost/benefits scheme, BIRD got close but was not chosen. The other potential solutions would require more work and support fewer requirements. With ExaBGP as the core of the new RIS system, we could create a new RIS system that allows us to support the use cases and the requirements that we have now, but also those that may come up in coming years. The idea is to use ExaBGP and add messages to a queueing system where ordering could be guaranteed (since ordering is really important). Different applications could consume the messages from the queue and process the data. Since the MRT-formatted files are used by the 'legacy' systems, it is required that the new system would also provide this feature. This is why an application needs to be developed to maintain the RIB state and create the MRT-formatted files, since this is not possible by only using ExaBGP. Another application would consume messages from the queue and insert the messages in HBase (so that applications can consult HBase for gathering data).

It is outlined in the research paper, added as an attachment, that it is recommended to develop a prototype. This prototype, which is similar to a proof of concept, would be used to see if it is indeed possible to meet all the requirements with the custom solution, based on ExaBGP. An high level overview of the prototype is showed below.

Figure 2: Design philosophy with ExaBGP as the RRC

We will publish another RIPE Labs article about the development and deployment stage of this project. It will outline the different components in the prototype.

Appendix A: List of requirements (MoSCoW scheme)

Appendix B: Research report

Research paper: Research on RIS Route Collectors by Wouter Miltenburg