New and Improved RIPE Registry Global Resource Service

Introduction and Background

For many years, the RIPE Database could be queried for non-RIPE registry data and non-RIPE routing information. The RIPE Routing Registry is also part of the Internet Routing Registry - a collection of databases holding global routing information, many of which mirror each other. One of the benefits of this new service described here is that all the data queried from  the RIPE Database sources is presented in RIPE RPSL format.

Currently the RIPE Database contains data from the following sources:

• RIPE
• RADb
• NTTCOM
• APNIC
• ARIN
• ARIN-RR
• JPIRR
• AfriNIC

You can find this information by sending the following query to the database:

  whois -q sources
 

Over time, the structure of the data that we import has changed and so have some of the referential integrity rules of the RIPE Database. The effect is that a lot of the imported data became stale. This problem increased over time because “unrecognised” objects or objects that broke referential integrity rules were removed. The RIPE NCC had to frequently reload the data sources from new "data dumps" to try to bring it back into sync with the authoritative data. But even during these reloads, unrecognised objects would still be dropped.

New Service - RIPE Registry Global Resource Service (GRS)

At the past two RIPE Meetings, RIPE 59 and 60, the RIPE NCC was asked to improve the quality of the alternative sources in the RIPE Database and to get back into sync with the authoritative data sources.

We decided to completely rewrite the service and have renamed it the RIPE Registry Global Resource Service (GRS). The data we present now is fully synchronised with the authoritative sources in a more resilient way. We have significantly improved the data transformation algorithms to handle the current data and future deviations. We also improved error logging and monitoring for cases where we detect a difference between the authoritive data and the data in the RIPE Database this way we can act upon these cases and keep this data accurate.

So far we have (re)established our agreements for access to bulk operational data for the following sources:

• RADb
• APNIC
• ARIN
• AfriNIC
• LACNIC

Please note that the LACNIC data will be included for the first time. This means the RIPE Database will contain the most complete set of operational data in (RIPE) RPSL format that has ever been available in one place.

So far, we have implemented the new GRS for data from:

• RADb
• APNIC
• ARIN
• LACNIC

   whois -q sources
  

   RIPE
RADB-GRS
APNIC-GRS
LACNIC-GRS
ARIN-GRS
NTTCOM
ARIN-RR
JPIRR
AFRINIC-GRS
RIPE-GRS
  

What Data is Part of the GRS and What is Done With the D ata Received?

Because the RIPE NCC is bound by Dutch and European data privacy laws, we are obliged to remove all personal data received from other databases. This is either removed at the source or stripped out and deleted during the transformation process. The RIPE NCC does not store any personal data from other registries. Where necessary, we create and reference dummy objects to keep data integrity intact.

RPSL Syntax

Before importing the data we transform objects into  RIPE RPSL syntax  by carrying out the following steps:

• Adding missing mandatory attributes
• Wrapping unrecognised attributes with "remarks"
• Creating dummy objects for missing data to keep referential integrity
• Converting attribute values
• All these transformations are marked by "End Of Line" comments in the objects

List of End Of Line Comments

Below is a list of the comments that get added during the transformation process together with an explanation of what they mean:

# Mirror: Removed private data

Some attribute values will be replaced by dummy data. For example “author:”, “tech-c:”, “admin-c:”.

# Mirror: Dummy object

Some dummy objects will be generated to conform to RIPE RPSL referential integrity rules. For example PERSON, ROLE, RTR-SET, AS-SET and ROUTE-SET. References to these dummy objects will be added where necessary.

# Mirror: Mandatory data

Dummy values created for missing mandatory attributes to conform with RIPE RPSL syntax.

# Mirror: Invalid reference

A reference which is not valid was removed. For example an attribute with a value containing a space or “@” which makes the reference invalid.

# Mirror: Added reference

In RIPE RPSL a ROUTE, ROUTE6, INET-RTR and AUT-NUM object can be a member of a set object by means of a “member-of:” attribute. Whenever the set object doesn't have a reference to the maintainer of the member object in the “mbrs-by-ref:” attribute then we generate such a reference.

# Mirror: Normalized pkey

The primary key of ROUTE6 and INET6NUM objects are normalised to the shortest valid form.

# Mirror: Empty attribute

Replace the attribute with a “remarks:” attribute if it has no actual value.

# Mirror: Unknown attribute

Any unknown attributes in an object type are wrapped by a “remarks:” attribute. The original, unrecognised attribute and value are still present in the remark.

# Mirror: Changed object type

This indicates the object-type in the source data had to be changed.

When data is returned from other sources it is stated the data is subject to the RIPE Database Terms and Conditions. This should properly reflect the Terms and Conditions or Copyright of the authoritative source.

We will be integrating the remaining data sources into the GRS shortly. We will also be exposing this data via our RESTful web services and new search clients.