You are here: Home > Publications > RIPE Labs > Paul Palse > New and Improved RIPE Registry Global Resource Service

New and Improved RIPE Registry Global Resource Service

Paul Palse — 21 Mar 2011
We have redesigned and improved the way we mirror other databases. We now have a method of translating the operational data from other registries (for instance from other Regional Internet Registries or the RADb) into the RIPE Database structure.

Note that this is an updated version of an earlier article with the same title and URL.

Introduction and Background

For many years, the RIPE Database could be queried for non-RIPE registry data and non-RIPE routing information. The RIPE Routing Registry is also part of the Internet Routing Registry - a collection of databases holding global routing information, many of which mirror each other. One of the benefits of this new service described here is that all the data queried from  the RIPE Database sources is presented in RIPE RPSL format.

Currently the RIPE Database contains data from the following sources:

  • RIPE
  • RADb
  • NTTCOM
  • APNIC
  • ARIN
  • ARIN-RR
  • JPIRR
  • AfriNIC

You can find this information by sending the following query to the database:

  whois -q sources
 

Over time, the structure of the data that we import has changed and so have some of the referential integrity rules of the RIPE Database. The effect is that a lot of the imported data became stale. This problem increased over time because “unrecognised” objects or objects that broke referential integrity rules were removed. The RIPE NCC had to frequently reload the data sources from new "data dumps" to try to bring it back into sync with the authoritative data. But even during these reloads, unrecognised objects would still be dropped.

New Service - RIPE Registry Global Resource Service (GRS)

At the past two RIPE Meetings, RIPE 59 and 60, the RIPE NCC was asked to improve the quality of the alternative sources in the RIPE Database and to get back into sync with the authoritative data sources.

We decided to completely rewrite the service and have renamed it the RIPE Registry Global Resource Service (GRS). The data we present now is fully synchronised with the authoritative sources in a more resilient way. We have significantly improved the data transformation algorithms to handle the current data and future deviations. We also improved error logging and monitoring for cases where we detect a difference between the authoritive data and the data in the RIPE Database this way we can act upon these cases and keep this data accurate.

So far we have (re)established our agreements for access to bulk operational data for the following sources:

  • RADb
  • APNIC
  • ARIN
  • AfriNIC
  • LACNIC

Please note that the LACNIC data will be included for the first time. This means the RIPE Database will contain the most complete set of operational data in (RIPE) RPSL format that has ever been available in one place.

So far, we have implemented the new GRS for data from:

  • RADb
  • APNIC
  • ARIN
  • LACNIC

The command:

   whois -q sources
  

now returns:

   RIPE
RADB-GRS
APNIC-GRS
LACNIC-GRS
ARIN-GRS
NTTCOM
ARIN-RR
JPIRR
AFRINIC-GRS
RIPE-GRS
  

 

As you can see "-GRS" has been added to the GRS sources.

Several new features added to Whois to support GRS

The following functionality was added:

  • In order to maintain backwards compatibility we map the source prefix to the corresponding '*-GRS' source. So querying for source 'ARIN' will result in a query to 'ARIN-GRS'

  • Searching on source GRS searches all '*-GRS' sources.

  • Queries on GRS sources are now exempt of accounting and filtering as no personal data is returned.

  • RIPE-GRS has been added to have a consistent output of all GRS sources.

What Data is Part of the GRS and What is Done With the D ata Received?

Because the RIPE NCC is bound by Dutch and European data privacy laws, we are obliged to remove all personal data received from other databases. This is either removed at the source or stripped out and deleted during the transformation process. The RIPE NCC does not store any personal data from other registries. Where necessary, we create and reference dummy objects to keep data integrity intact.

RPSL Syntax

Before importing the data we transform objects into  RIPE RPSL syntax  by carrying out the following steps:

  • Adding missing mandatory attributes
  • Wrapping unrecognised attributes with "remarks"
  • Creating dummy objects for missing data to keep referential integrity
  • Converting attribute values
  • All these transformations are marked by "End Of Line" comments in the objects

List of End Of Line Comments

Below is a list of the comments that get added during the transformation process together with an explanation of what they mean:

# Mirror: Removed private data

Some attribute values will be replaced by dummy data. For example “author:”, “tech-c:”, “admin-c:”.

# Mirror: Dummy object

Some dummy objects will be generated to conform to RIPE RPSL referential integrity rules. For example PERSON, ROLE, RTR-SET, AS-SET and ROUTE-SET. References to these dummy objects will be added where necessary.

# Mirror: Mandatory data

Dummy values created for missing mandatory attributes to conform with RIPE RPSL syntax.

# Mirror: Invalid reference

A reference which is not valid was removed. For example an attribute with a value containing a space or “@” which makes the reference invalid.

# Mirror: Added reference

In RIPE RPSL a ROUTE, ROUTE6, INET-RTR and AUT-NUM object can be a member of a set object by means of a “member-of:” attribute. Whenever the set object doesn't have a reference to the maintainer of the member object in the “mbrs-by-ref:” attribute then we generate such a reference.

# Mirror: Normalized pkey

The primary key of ROUTE6 and INET6NUM objects are normalised to the shortest valid form.

# Mirror: Empty attribute

Replace the attribute with a “remarks:” attribute if it has no actual value.

# Mirror: Unknown attribute

Any unknown attributes in an object type are wrapped by a “remarks:” attribute. The original, unrecognised attribute and value are still present in the remark.

# Mirror: Changed object type

This indicates the object-type in the source data had to be changed.

Known Issues

When data is returned from other sources it is stated the data is subject to the RIPE Database Terms and Conditions. This should properly reflect the Terms and Conditions or Copyright of the authoritative source.

Next Steps

We will be integrating the remaining data sources into the GRS shortly. We will also be exposing this data via our RESTful web services and new search clients. 

2 Comments

Richard says:
21 Oct, 2011 09:07 PM
How do i report abuse from scammer registered with your data base Hiking my computer
Denis Walker says:
24 Oct, 2011 10:47 AM
HI Richard

We have a list of Frequently Asked Questions about abuse including hacking that you may find useful

http://www.ripe.net/data-tools/db/faq/faq-hacking-spamming

We also have a web based tool for finding abuse contacts if you know the IP address used by a hacker

https://labs.ripe.net/Members/Paul_P_/content-abuse-finder

Regards
Denis Walker
Business Analyst
RIPE NCC Database Group
Add comment

You can add a comment by filling out the form below. Comments are moderated so they won't appear immediately. If you have a RIPE NCC Access account, we would like you to log in.