We have redesigned and improved the way we mirror other databases. We now have a method of translating the operational data from other registries (for instance from other Regional Internet Registries or the RADb) into the RIPE Database structure.
Note that this is an updated version of an earlier article with the same title and URL.
Introduction and Background
For many years, the RIPE Database could be queried for non-RIPE registry data and non-RIPE routing information. The RIPE Routing Registry is also part of the Internet Routing Registry - a collection of databases holding global routing information, many of which mirror each other. One of the benefits of this new service described here is that all the data queried from the RIPE Database sources is presented in RIPE RPSL format.
Currently the RIPE Database contains data from the following sources:
You can find this information by sending the following query to the database:
whois -q sources
Over time, the structure of the data that we import has changed and so have some of the referential integrity rules of the RIPE Database. The effect is that a lot of the imported data became stale. This problem increased over time because “unrecognised” objects or objects that broke referential integrity rules were removed. The RIPE NCC had to frequently reload the data sources from new "data dumps" to try to bring it back into sync with the authoritative data. But even during these reloads, unrecognised objects would still be dropped.
New Service - RIPE Registry Global Resource Service (GRS)
At the past two RIPE Meetings, RIPE 59 and 60, the RIPE NCC was asked to improve the quality of the alternative sources in the RIPE Database and to get back into sync with the authoritative data sources.
We decided to completely rewrite the service and have renamed it the RIPE Registry Global Resource Service (GRS). The data we present now is fully synchronised with the authoritative sources in a more resilient way. We have significantly improved the data transformation algorithms to handle the current data and future deviations. We also improved error logging and monitoring for cases where we detect a difference between the authoritive data and the data in the RIPE Database this way we can act upon these cases and keep this data accurate.
So far we have (re)established our agreements for access to bulk operational data for the following sources:
Please note that the LACNIC data will be included for the first time. This means the RIPE Database will contain the most complete set of operational data in (RIPE) RPSL format that has ever been available in one place.
So far, we have implemented the new GRS for data from:
whois -q sources
RIPE RADB-GRS APNIC-GRS LACNIC-GRS ARIN-GRS NTTCOM ARIN-RR JPIRR AFRINIC-GRS RIPE-GRS
As you can see "-GRS" has been added to the GRS sources.
Several new features added to Whois to support GRS
The following functionality was added:
In order to maintain backwards compatibility we map the source prefix to the corresponding '*-GRS' source. So querying for source 'ARIN' will result in a query to 'ARIN-GRS'
Searching on source GRS searches all '*-GRS' sources.
Queries on GRS sources are now exempt of accounting and filtering as no personal data is returned.
- RIPE-GRS has been added to have a consistent output of all GRS sources.
What Data is Part of the GRS and What is Done With the D ata Received?
Because the RIPE NCC is bound by Dutch and European data privacy laws, we are obliged to remove all personal data received from other databases. This is either removed at the source or stripped out and deleted during the transformation process. The RIPE NCC does not store any personal data from other registries. Where necessary, we create and reference dummy objects to keep data integrity intact.
Before importing the data we transform objects into RIPE RPSL syntax by carrying out the following steps:
- Adding missing mandatory attributes
- Wrapping unrecognised attributes with "remarks"
- Creating dummy objects for missing data to keep referential integrity
- Converting attribute values
- All these transformations are marked by "End Of Line" comments in the objects
List of End Of Line Comments
Below is a list of the comments that get added during the transformation process together with an explanation of what they mean:
# Mirror: Removed private data
Some attribute values will be replaced by dummy data. For example “author:”, “tech-c:”, “admin-c:”.
# Mirror: Dummy object
Some dummy objects will be generated to conform to RIPE RPSL referential integrity rules. For example PERSON, ROLE, RTR-SET, AS-SET and ROUTE-SET. References to these dummy objects will be added where necessary.
# Mirror: Mandatory data
Dummy values created for missing mandatory attributes to conform with RIPE RPSL syntax.
# Mirror: Invalid reference
A reference which is not valid was removed. For example an attribute with a value containing a space or “@” which makes the reference invalid.
# Mirror: Added reference
In RIPE RPSL a ROUTE, ROUTE6, INET-RTR and AUT-NUM object can be a member of a set object by means of a “member-of:” attribute. Whenever the set object doesn't have a reference to the maintainer of the member object in the “mbrs-by-ref:” attribute then we generate such a reference.
# Mirror: Normalized pkey
The primary key of ROUTE6 and INET6NUM objects are normalised to the shortest valid form.
# Mirror: Empty attribute
Replace the attribute with a “remarks:” attribute if it has no actual value.
# Mirror: Unknown attribute
Any unknown attributes in an object type are wrapped by a “remarks:” attribute. The original, unrecognised attribute and value are still present in the remark.
# Mirror: Changed object type
This indicates the object-type in the source data had to be changed.
When data is returned from other sources it is stated the data is subject to the RIPE Database Terms and Conditions. This should properly reflect the Terms and Conditions or Copyright of the authoritative source.
We will be integrating the remaining data sources into the GRS shortly. We will also be exposing this data via our RESTful web services and new search clients.