This project was initiated to research, develop and deploy a prototype that could potentially replace the current Remote Route Collectors (RRCs) and which invokes less delay. It was a graduation project for my bachelor thesis at the HvA (Hogeschool van Amsterdam). This article outlines the possible solutions that we found and examined during the research stage of this project. It also describes a proposal to create a prototype, based on the chosen solution.
Note: The full research report can be found in Appendix B .
Please also see the second part of this project:
Building the Next Generation RIS Collectors.
Introduction
The Routing Information Service (RIS) , provided by the RIPE NCC, was established in 2001 and started with the idea to collect and store Internet routing data from several locations around the globe. This data, which is collected and saved by the RIS Remote Route Collectors, can be accessed via RIPEstat and used for further analyses. A Remote Route Collector (RRC), as the name suggests, collects all the Internet routing data from several points around the world and stores this data for further processing. This data is saved in the MRT Routing Information Export format and used by different applications, both by the community and internal RIPE NCC systems.
The current implementation runs in a batch-oriented mode that involves a lot of delay. Even though there are new requirements, the system still relies on the data collected by the RRCs. Therefore we need to look at possible alternatives that could be used to collect and store the Internet routing data. Before it is even possible to look at the possible alternatives that exist, we need to first look at the current requirements.
Requirements
During the first few weeks of the internship, I found out which requirements currently exist. New feature requests from the community regarding the RRCs have been added to the requirements list.
Now that the requirements are known, it would be useful to prioritise them. We can give each requirement a priority using the MoSCoW method . After talking to my colleagues at the RIPE NCC, we agreed on the final MoSCoW scheme, which was used throughout the entire project. You can find the complete list of requirements (based on the MoSCow scheme) in Appendix A .
One of the most important requirements is the requirement to maintain the Multi-threaded Routing Toolkit (MRT) formatted files for backwards compatibility. These files are currently used by the community and internal systems of the RIPE NCC to process the data. The MRT Routing Information Export Format is the de facto standard for storing Internet routing data. Since all processes rely on these files, the next generation of RIS should also facilitate this.
There is one other requirement that is very important, and that is also one of the reasons why this project has been initiated. It is the requirement for new RRCs to allow for lower delays between the data collection and the visibility in the datasets. The goal of the new RRCs is to allow future implementation to gain less delay with the overall RIS system. It involves changes that have to be made in the overall processing pipeline.
Potential Solutions
An application or project was considered a 'potential solution' when it could meet at least one of the requirements. The software solutions covered below are all considered to be within the scope of this research project.
ExaBGP
ExaBGP , the "BGP swiss army knife of networking" (and I now definitely concur with that), was considered a potential solution. Since we would like to move from the batch-oriented process to something more stream-oriented or event-driven, ExaBGP could be used as the core of the whole prototype and provide other applications with the data that it receives. Unfortunately it does not maintain the RIB state, which could also be one of ExaBGP's strengths since it provides you with the information that you want and any further processing depends on your application. This gives you the freedom to use ExaBGP in many use cases.
Since it does not maintain a RIB state, another application should provide this feature. But which application? The goal is to develop a modular, extensible solution that does not only support the current requirements, but also allows for future changes. The illustration below describes this philosophy.
Figure 1: Data flow and design philosophy of the prototype
BGPmon
BGPmon meets our requirements, with one limitation: It is not possible to announce routes with BGPmon , which implies that we cannot meet one of the "must-have" requirements. It is a design decision of the BGPmon team, so it will quite likely never be supported by BGPmon. This puts a serious limitation on the use of BGPmon. We could modify BGPmon to add support for announcements, but even then the changes will never make it into the mainstream distribution, which means we would have to maintain it.
OSR Quagga
This is a Quagga version maintained by the OSR group and it pushes their modifications to the mainstream Quagga release. Their Github page also explicitly tells visitors to use the regular Quagga releases. Ignoring this warning and cloning their repository, I could not find any changes that would make this version more usable than the mainstream Quagga releases.
BIRD
BIRD performs faster than Quagga, as was outlined in a presentation at NANOG 48 following presentation . If it is decided to use BIRD, we would basically end up with the same approach as we are using now. It could be faster and maybe allows us to connect with more peers, but when you look at the requirements, you see that we want more than only having better performance.
OpenBGPD
OpenBGPD also performs faster than Quagga, as you can see in the presentation referred to above. However, as with BIRD, we would still end up with the same implementation.
XORP
The eXtensible Open Router Platform sounds like a solution which could provide us with the features we want. It is stated that XORP is designed to be secure and, even in the event of a process failure, other processes should not be affected. If we want to create a real extensible and modular application that relies on the data of XORP, more time would be required to develop the applications than with the other alternatives.
Vyatta
If you want to have a firewall, router, VPN, and traffic management system Vyatta could be an interesting solution. However, looking at the requirements that we have, we want to have access to the BGP messages. Moreover, Quagga is used at the core for routing, so it would again be a same approach that we are using right now, since Quagga would create the MRT-formatted files.
PyRT
Python Routing Toolkit is fairly old, so old that their website certificate has expired (it also has an invalid root certificate). It is only a BGP listener and does not send any messages. You might ask, is it the same problem as with BGPmon? Yes, but it also does not send any keepalive messages, which results in peers terminating the connections after some time.
BMP
The BGP monitoring protocol is a protocol that really suits our use cases. The only disadvantages right now are that it is still a draft and that all neighbours have to support BMP. We want to be able to connect to as many neighbours as possible and this will be a real restriction in choosing our peering neighbours. The other downside is, since the protocol is relatively new and still a draft, not many applications support this protocol. It could be the case that we have to develop a server application for storing and processing the messages. The concept is really good, all the messages stored in the Adj-RIB-In of the neighbour would be send to the monitoring stations which will send the messages to the server for further processing. The messages are encapsulated in Route Monitoring messages and must be parsed by another application (which must be developed to meet our use cases).
Ryu
Ryu is a component-based software-defined networking framework. It is a framework that could be used to receive BGP updates and to do further processing upon this data. In order to support the use-cases that we have, an application needs to be developed that uses the Ryu framework. Ryu itself uses multiple processes and threads to provide all the functionalities to the application that is using this framework. It is possible to use the framework to encode and decode the BGP packets. The only downside is that an application needs to be developed that maintains the BGP connections and the state. It would involve a lot more work then the applications described previously, since they provide the same functions and already maintain the BGP connection.
Making a Decision
Now that we know what all the potential solutions are capable of, a cost/benefits scheme was created. In our case, the cost was the time that needed to be invested in extending the feature set or modifying some parts of the potential solution. A relatively high score means that less to no work is involved in meeting all the requirements, whereas a relatively low score means that a lot of work is involved in meeting the requirements. We also examiend whether it was realistic to choose a certain 'potential solution' and in the attached research paper you can read how we have made the decision.
Conclusion
ExaBGP came out on top if you look at the total score of the cost/benefits scheme, BIRD got close but was not chosen. The other potential solutions would require more work and support fewer requirements. With ExaBGP as the core of the new RIS system, we could create a new RIS system that allows us to support the use cases and the requirements that we have now, but also those that may come up in coming years. The idea is to use ExaBGP and add messages to a queueing system where ordering could be guaranteed (since ordering is really important). Different applications could consume the messages from the queue and process the data. Since the MRT-formatted files are used by the 'legacy' systems, it is required that the new system would also provide this feature. This is why an application needs to be developed to maintain the RIB state and create the MRT-formatted files, since this is not possible by only using ExaBGP. Another application would consume messages from the queue and insert the messages in HBase (so that applications can consult HBase for gathering data).
It is outlined in the research paper, added as an attachment, that it is recommended to develop a prototype. This prototype, which is similar to a proof of concept, would be used to see if it is indeed possible to meet all the requirements with the custom solution, based on ExaBGP. An high level overview of the prototype is showed below.
Figure 2: Design philosophy with ExaBGP as the RRC
We will publish another RIPE Labs article about the development and deployment stage of this project. It will outline the different components in the prototype.
Appendix A: List of requirements (MoSCoW scheme)
Must | |
Announcements of anchor/beacon IP | |
MRT-formatted files | |
RAW data | |
Metadata | |
Ordering | |
Less delay | |
Same attributes | |
eBGP multihop | |
Scaling | |
Should | |
Live data stream | |
Integrity of data | |
Extensible | |
Could | |
High resolution timestamp | |
Authorisation and encryption | |
Would | |
None | |
Out of scope | |
Spike detection | |
Aggregation of data | |
Correlation between RIPE Atlas/RIPE Atlas anchors and RRC data | |
Separate full feed from partial feed |
Appendix B: Research report
Research paper: Research on RIS Route Collectors by Wouter Miltenburg
Comments 7
Comments are disabled on articles published more than a year ago. If you'd like to inform us of any issues, please reach out to us via the contact form here.
Rico Glöckner •
the appendix-Link takes me straight to access.ripe.net/profile. Not sure what i'm doing wrong.
Hide replies
Mirjam Kühne •
Rico, Thanks for letting us know. This is fixed. You should be able to view the pdf now. Sorry about this.
Wilfried Woeber •
I'm wondering (and haven't read the full paper yet) is there anything that relates to RPKI?
Hide replies
Wouter Miltenburg •
Wilfried, in the paper I didn't mention RPKI. If I recall correctly, RPKI is entirely an out-of-band system, and doesn't involve changes in the BGP update messages or the RRCs' supported capabilities. If so, the RRCs are not involved in RPKI, since it receives the messages and relays them to other systems. The RRCs should accept all the messages and relay it to the back-end. Such an approach allows us that future requirements, which could be the validation of the update messages received by an RRC, to be supported by the new RRC system as well. It would involve changes/developing applications, but the RRC system would be the same (if no new capabilities need to be supported by the RRC itself). If it is decided that the BGP update messages would be validated using RPKI, it would be done in a later stage of the whole processing pipeline. It would be a post-analysis task and the scope of this project was primarily the data gathering/collection part. Please correct me if I am wrong, if there is something we need to add into the collectors related to RPKI.
JK •
I'm curious if you could take a slightly different approach here. If you can get the IX operator involved, it has some huge side benefits. In that case, then I think you would be at least doing BIRD development, or both BIRD and OpenBSD. I would argue that IX operators are also interested in real-time BGP update captures and integrity for playback. And that having them involved is the most efficient vantage point, as well as from a management perspective.
rg •
Thoughts: 1) I don't see how involvement of IX Operators would be beneficial at all, the BGP Sessions can be multi hop and don't need any geographic or topographic relation. Relaying BGP Feeds from an IX RR doesn't help also, since its mostly also sending the FIB, not individual RIBs. Topological relation would only help stability of feeds, IMHO. 2) RPKI doesn't belong in any RS indeed, any checking can be done in the UI (as it is don't today already inside the lirportal) 3) i don't see why you would think multithreading to be required, indeed, theres not even an requirement to run all BGP Feeds inside one Process - on the contrary it might have benefits to run every feed in its own process (the only two takeaways are configuration simplicity and possible limitations in bind(), which then again might be workarounded, too). If you run several Feeds inside one Process you won't get along without Multithreads, as have been proven in the past. 4) Any Routecollecting Process can be done in a scripting language, and probably should be done. There are BGP Parsing (whistle) Libraries in almost any Language, and with observers (or call/hookbacks) available in most languages this should actually be a really easy task. And even if there was no such thing yet, it might be beneficial for the community if RIPE NCC would support such a project actively. 5) You should take great care for choosing a Messaging Bus, while BGP itself is pretty lightweight traffic wise, transmitting the data in any other protocol might severely enlarge the data stream, which gets worse over distance. 6) it might be beneficial to keep the RIB for a BGP Session on the remote side of the message bus, but you need to keep it; because for example if a peer shuts down its session, you need to flush the RIB (i.e. inject pseudo-Withdrawals in your Datastream). The Question is where exactly you need to see those, depending on that you might also need to keep the RIB in the respective route-collecting process. Umm, from the top of my head, if my memory serves me more or i see something in the PDF (haven't read it yet), i will add it. -rg
Hide replies
Wouter Miltenburg •
Thanks for your comments. We will take this into account. Please also read the article that we will publish about the development and deployment stage of this topic.