RIPE NCC NetSense

Perhaps of more interest to this audience, the space shuttle Discovery launched the Hubble space telescope into orbit, while in Redmond, Microsoft released Windows 3.0, and at CERN, Tim Berners Lee published his ‘Proposal for a HyperText Project’. What is perhaps less well known is that in October of 1990, RIPE (before the NCC even existed), published the very first DNS host count of Internet connected hosts in Europe. Thus began a long history of Internet measurement projects which the NCC continues to support and develop almost 20 years later.

I’ve been responsible for most of these projects for the past three years, and in this post, I’ll explain some of the changes and developments which have happened in that time, culminating in the release of our new prototype service, RIPE NCC NetSense http://netsense.ripe.net  

After the Hostcount

In 1999, my predecessors launched the Routing Information Service, or RIS - a huge data store of BGP routing data collected from (nowadays) 16 diverse global locations. We usually refer to this as a looking glass with history, and it’s among our most popular services, giving network operators a unique insight to the visibility and stability of their prefixes over time. The largest issue with RIS is scaling the back-end to cope with the sheer volume of data we hold - the database stands at just over 1Tb as I write. 

Shortly after RIS, in 2000, we launched the Test Traffic Measurements (TTM) service which enables users (with the aid of special hardware) to carry out incredibly accurate monitoring of their connectivity to other parts of the Internet in realtime. TTM was revolutionary at launch, but with a heavy scientific focus and non-obvious consumer appeal, the network never achieved the necessary rapid organic growth required to realise its full potential.

More recently, in 2004, our DNS Monitoring service, DNSMON, began to monitor the K-root infrastructure from a globally distributed system of measurement probes. The benefits of this level of detailed operational monitoring proved to be enormous, and the service was extended to monitor the entire root server infrastructure, as well as being opened as a subscription service to ccTLD, gTLD and ENUM operators.

This suite of services have much in common, augmenting and supporting each other in many areas - DNSMON, for example, is based on the TTM network, and RIS can be used to examine connectivity issues highlighted by TTM. In spite of these obvious synergies, the services, which were developed as standalone applications, remained isolated from each other after five or so years of operation. With the limitations of this legacy became ever more painful, the time for integration was ripe.

Recent history

We knew our users could derive huge benefits from our services being operated and maintained by a single team, and so when the RIPE NCC was re-organised in 2006, the Information Services team (or IS) was formed and we began the difficult task of uniting our services.

One of my first observations was that our stuff was, at a basic level, actually rather good - we had a large measurement network (although pretty Euro-centric) and heaps of useful data - but confused interfaces and disappointing user numbers. We tried to improve this in various ways, including: 

• marketing - taking IS to conferences, LIR training courses, IXP meetings, NOG meetings - to increase our exposure and get more user feedback
• streamlining and simplifying TTM pricing and sales
• back-end improvements - new hardware to improve speed and resilience
• incremental feature improvements - simple enhancements, cosmetic and usability fixes
• new features - the RIS dashboards, a new interface for Hostcount, etc
• expanding the measurement network:
  • promoting TTM to end users and encouraging take-up
  • sponsored RIS route collector and TTM deployments in significant locations
  • strategic partnerships to gain access to the regions - MoU’s are in place with APNIC, LACNIC and AfriNIC

Our efforts had pleasing and measurable success - we spread the network further across Asia and into Africa, increased usage, and made new fans with our new features, but the ‘wow’ factor continued to elude us, and our services remained unexciting.

During this time, the RIPE NCC began to focus on our three strategic pillars - resource lifecycle management, evolving our role, and trusted source of data. With Information Services underpinning the NCC’s recognition as a trusted source of data, we realised we’d have to scale up our activities a little, and that we lacked the skills to do that independently.

Outside help

At the end of 2008, we began to work with an innovation consultancy called SunIdee, conveniently located just a few minutes walk from the NCC offices in Amsterdam.

Through a series of internal workshops and interviews, SunIdee helped us to catalogue our entire portfolio (more than 30 individual tools and services, some of which were unused or unmaintained), and exhibited to us that many tools were so broad in scope, that most users had no idea what to do with them. This was a revelation for us. We also brainstormed to estimate what our users were interested in and how we could offer more value to them.

Next, we set about defining our stakeholders. Traditionally, we’d positioned our services solely at technical audiences - ISP engineers, academic researchers and so on - but the environment has changed since 1990, and nowadays we have governments, regulators, law enforcement agencies and many others interested in the data we produce. Obviously, the majority of our membership come from the technical community, so although this group retains our focus, we have begun to consider others.

During RIPE 58 in Amsterdam in May 2009, SunIdee helped us to conduct stakeholder interviews and workshops with more than 30 members of our community. We focused on validating the assumptions we’d made earlier in the process, and captured far more new suggestions, ideas and feature requests than we’d ever imagined. A daunting amount of work, but promising nonetheless. This was the first time we’d put so much effort into connecting directly with our users, and the experience was overall an incredibly positive one. 

The output from this really simplified our approach to the delivery of services to our stakeholders. We put in place a stucture which would enable us to position services in a sensible and intuitive manner, by considering two different axes. On the vertical axis, we focus on complexity - data, in its rawest form, followed by tools to query that data, followed by analysis based on the output from the tools. On the horizontal axis, we looked more closely at the user needs - monitor, for real time monitoring, diagnose, for looking back in time to identify issues, and forecast, for future insights. These combine to form the grid to the right.

The next steps were actually quite simple - we looked at our stakeholder needs alongside our existing portfolio, and worked on a three-pronged development plan. Firstly, we looked at quick wins - simple changes we could make to our tools to increase their usability and usefulness. Secondly, some medium term goals - additional richness we could draw from our existing data sets by investing some development time. Thirdly and finally, the really revolutionary stuff - long term development requiring significant additions or changes to our entire infrastructure.

RIPE NCC NetSense

NetSense emerged from the medium term developments, as the next generation of our information gathering, analysis and presentation. We were very eager to have something in place prior to RIPE 59, so we needed to re-think our development strategy. Historically we’d used the fairly traditional waterfall model of project management, but we wanted something a bit more flexible, and settled on agile and scrum, as they were already successfully in use by other development teams at the NCC.

Without going into too much detail here (there are plenty of other great articles on this topic online), scrum involves (among other things) splitting work up into short 2-week ‘sprints’. I set the sprint goals, but my team has the freedom to choose how to implement them, and an explicit agreement that goals won’t shift mid-sprint. At the end of each sprint, we arrange a demo of the completed work, and any feedback can be worked into the next sprint. In this way, NetSense was gradually and iteratively developed over the course of the past few months.

The philosophy behind NetSense is to produce a single web application, with a rich user interface, which we can use to present data from our existing services (RIS, TTM etc) in a more intuitive and simple manner than ever before, so that we can provide a single point to query all of our data, and eventually dispose of the many legacy tools. It’s been designed to be flexible and easily extendable so that we can build on it over time, by adding in new views of data combined from different sources, or data from entirely new sources.

In this first release, NetSense provides a global connectivity health overview, and allows network operators to input their AS or prefix to move to an interactive AS dashboard where they can check visibility and routing consistency and other interesting data. Functionality is limited at present - we put a lot of resource into getting the site looking just right - but now that it does, our energies can be directed solely at adding new features.

To ensure that we do this in the best way possible, we’re actively encouraging feedback. You can provide feedback through the NetSense site itself at http://netsense.ripe.net , via email to netsense@ripe.net , or by joining in the discussion on RIPE Labs.

Summary

NetSense is just one of the first wave of new tools and applications to have come from the RIPE NCC, and I’m really pleased that we’ve been able to unveil this using RIPE Labs. In my three years at the NCC, this is the biggest and boldest move made by Information Services to get up close and personal with our membership and our community.

I was so enthused by the level of input and interest shown by our users when we talked to them about how we could improve our suite of Information Services, and I hope you’re pleased with the output. But the work doesn’t stop here - development of these all of our tools is ongoing, and via RIPE Labs, you can participate with the process almost in real time in a way which has never really been available before.

I, or one of my team, will come back to RIPE Labs fairly regularly in search of feedback and input, and when we have new functionality or tools to talk about, you’ll read it here first. If you can’t keep up, then be sure to look out for us at RIPE 60!