A while ago we announced the new Global Resource Service (GRS) for the RIPE Database. It allows you to query other Regional Internet Registry (RIR) databases and Routing Registry data using the RIPE Database interface. In this article we describe this service some more and show some examples.
Note that this article is not up to date anymore. Please refer to the most up-to-date RIPE Databse documentation .
Introduction
A while ago we announced the new Global Resource Service (GRS) for the RIPE Database. It allows for queries to other Regional Internet Registry (RIR) databases and Routing Registry data using the RIPE Database interface. The service will return RIPE RPSL formatted object. With the addition of these sources to our RESTful API, responses can be retrieved in XML and JSON using the same schema as the RIPE data. All features of the RESTful Query Web Services are also available for the two new GRS Query Web Service.
Unlimited queries
We also added a GRS version of the RIPE data to the other GRS sources so that RIPE data can be accessed in the same format. When using the GRS sources, there are no limitations on the amount of data that a client can query for.
REST services URLs and GRS sources
http://lab.db.ripe.net/whois/grs-lookup/<source>/<object type>/<primary-key>.xml
The service URL for the grs-search service is:
http://lab.db.ripe.net/whois/grs-search.xml?flags=<flags>&source=<source>&query-string=<query>
The "source" parameter can be one or more of the following :
| Source | Parameter |
|---|---|
| RIPE | source=ripe |
| RADb | source=radb |
| APNIC | source=apnic |
| ARIN | source=arin |
| AfriNIC | source=afrinic |
| LACNIC | source=lacnic |
Example:
http://lab.db.ripe.net/whois/grs-lookup/apnic/inetnum/193.0.0.0%20-%20193.255.255.255.xml
http://lab.db.ripe.net/whois/grs-search.xml?flags=&source=ripe&source=apnic&source=afrinic&source=arin&source=lacnic&query-string=as3333
Content negotiation
All the content negotiation features documented for the other query services (negotiation by URL extension and negotiation by Accept header mime type) are available also on these two services. For more details, please refer to the updated RIPE Database API documentation .
Search client
We also made the GRS functionality available through our search client form.
| Some GRS sources under construction Currently the RIPE source is not available. The AfriNIC source is not currently in true GRS format and the data is quite stale. We're working hard to resolve the issues. |
|---|
Some technical background
After publishing the first GRS article, we got some questions about what we do with the data we receive that turns it into GRS data.
The best way to explain this is by giving some examples:
Referential integrity and dummy objects
ASHandle: AS26666 OrgID: DOEORG-7 ASName: DOENET ASNumber: 26666 RegDate: 2009-04-23 Updated: 2009-04-23 NOCHandle: JDOE42-ARIN TechHandle: JDOE42-ARIN AbuseHandle: JDOE42-ARIN Source: ARIN
We construct an AUT-NUM object from this information and turn any unrecognised attributes into "remark:" lines:
aut-num: AS26666 # Mirror: Changed object type org: ORG-MIRR1-RIPE # Mirror: Removed private data as-name: DOENET remarks: ASNumber: 26666 # Mirror: Unknown attribute remarks: RegDate: 2009-04-23 # Mirror: Unknown attribute changed: bitbucket+whois-mirror@ripe.net 20090423 # Updated tech-c: MIRR-RIPE # Mirror: Removed private data source: ARIN descr: DUMMY DESCRIPTION # Mirror: Mandatory data admin-c: MIRR-RIPE # Mirror: Mandatory data mnt-by: MIRROR-RIPE-MNT # Mirror: Mandatory data
We replace any personal data with dummy data and in order to retain referential integrity we also create links to dummy objects.
person: MIRR-RIPE # Mirror: Dummy object nic-hdl: MIRR-RIPE # Mirror: Dummy object address: DUMMYLAND # Mirror: Mandatory data phone: +3866938669 # Mirror: Mandatory data changed: bitbucket+whois-mirror@ripe.net 00000000 # Mirror: Mandatory data source: ARIN # Mirror: Mandatory data mntner: MIRROR-RIPE-MNT # Mirror: Dummy object descr: DUMMY DESCRIPTION # Mirror: Mandatory data admin-c: MIRR-RIPE # Mirror: Mandatory data upd-to: bitbucket+whois-mirror@ripe.net # Mirror: Mandatory data auth: MD5-PW $1$222222222222222222222222222222/ # Mirror: Mandatory data mnt-by: MIRROR-RIPE-MNT # Mirror: Mandatory data referral-by: MIRROR-RIPE-MNT # Mirror: Mandatory data changed: bitbucket+whois-mirror@ripe.net 00000000 # Mirror: Mandatory data source: ARIN # Mirror: Mandatory data organisation: ORG-MIRR1-RIPE # Mirror: Dummy object org-name: ORGNAME # Mirror: Mandatory data org-type: OTHER # Mirror: Mandatory data address: DUMMYLAND # Mirror: Mandatory data e-mail: bitbucket+whois-mirror@ripe.net # Mirror: Mandatory data mnt-ref: MIRROR-RIPE-MNT # Mirror: Mandatory data mnt-by: MIRROR-RIPE-MNT # Mirror: Mandatory data changed: bitbucket+whois-mirror@ripe.net 00000000 # Mirror: Mandatory data source: ARIN # Mirror: Mandatory data
Normalisation of ranges
Again an object we received from ARIN:
V6NetHandle: NET6-2600-1000-1 OrgID: DOEINC Parent: NET6-2600-1 NetName: DOENET NetRange: 2600:1000:: - 2600:1017:FFFF:FFFF:FFFF:FFFF:FFFF:FFFF NetType: allocation RegDate: 2010-04-06 Updated: 2010-04-06 Source: ARIN
Because the RIPE Database doesn't support IPv6 ranges but only prefix notation, we split the original object into two prefixes:
inet6num: 2600:1000::/28 # Mirror: Changed object type remarks: V6NetHandle: NET6-2600-1000-1 # Mirror: Unknown attribute org: ORG-MIRR1-RIPE # Mirror: Removed private data remarks: Parent: NET6-2600-1 # Mirror: Unknown attribute netname: DOENET status: allocation remarks: RegDate: 2010-04-06 # Mirror: Unknown attribute changed: bitbucket+whois-mirror@ripe.net 20100406 # Updated source: ARIN descr: DUMMY DESCRIPTION # Mirror: Mandatory data country: ZZ # Mirror: Mandatory data admin-c: MIRR-RIPE # Mirror: Mandatory data tech-c: MIRR-RIPE # Mirror: Mandatory data mnt-by: MIRROR-RIPE-MNT # Mirror: Mandatory data inet6num: 2600:1010::/29 # Mirror: Changed object type remarks: V6NetHandle: NET6-2600-1000-1 # Mirror: Unknown attribute org: ORG-MIRR1-RIPE # Mirror: Removed private data remarks: Parent: NET6-2600-1 # Mirror: Unknown attribute netname: DOENET status: allocation remarks: RegDate: 2010-04-06 # Mirror: Unknown attribute changed: bitbucket+whois-mirror@ripe.net 20100406 # Updated source: ARIN descr: DUMMY DESCRIPTION # Mirror: Mandatory data country: ZZ # Mirror: Mandatory data admin-c: MIRR-RIPE # Mirror: Mandatory data tech-c: MIRR-RIPE # Mirror: Mandatory data mnt-by: MIRROR-RIPE-MNT # Mirror: Mandatory data
And the dummy references:
person: MIRR-RIPE # Mirror: Dummy object nic-hdl: MIRR-RIPE # Mirror: Dummy object address: DUMMYLAND # Mirror: Mandatory data phone: +3866938669 # Mirror: Mandatory data changed: bitbucket+whois-mirror@ripe.net 00000000 # Mirror: Mandatory data source: ARIN # Mirror: Mandatory data mntner: MIRROR-RIPE-MNT # Mirror: Dummy object descr: DUMMY DESCRIPTION # Mirror: Mandatory data admin-c: MIRR-RIPE # Mirror: Mandatory data upd-to: bitbucket+whois-mirror@ripe.net # Mirror: Mandatory data auth: MD5-PW $1$222222222222222222222222222222/ # Mirror: Mandatory data mnt-by: MIRROR-RIPE-MNT # Mirror: Mandatory data referral-by: MIRROR-RIPE-MNT # Mirror: Mandatory data changed: bitbucket+whois-mirror@ripe.net 00000000 # Mirror: Mandatory data source: ARIN # Mirror: Mandatory data organisation: ORG-MIRR1-RIPE # Mirror: Dummy object org-name: ORGNAME # Mirror: Mandatory data org-type: OTHER # Mirror: Mandatory data address: DUMMYLAND # Mirror: Mandatory data e-mail: bitbucket+whois-mirror@ripe.net # Mirror: Mandatory data mnt-ref: MIRROR-RIPE-MNT # Mirror: Mandatory data mnt-by: MIRROR-RIPE-MNT # Mirror: Mandatory data changed: bitbucket+whois-mirror@ripe.net 00000000 # Mirror: Mandatory data source: ARIN # Mirror: Mandatory data
Conclusion
We are in the process of bringing RIPE data into the GRS format and improving the AfriNIC source. Already the GRS service can be used for high load and frequency queries for network resources without the risk of getting blocked or throttled.
As always, we are very interested in your feedback.
Comments 5
Comments are disabled on articles published more than a year ago. If you'd like to inform us of any issues, please reach out to us via the contact form here.
Anonymous •
The GRS sources are a highly valued addition to the REST API!<br />Finally one can dump peval...<br />Two things stuck me:<br /><br />"RIPE" is not a valid GRS source (contrary to what the article says, making the example fail):<br /><br /><a href="http://lab.db.ripe.net/whois/grs-search.xml?flags=&source=ripe&source=apnic&source=afrinic&source=arin&source=lacnic&query-string=as3333" rel="nofollow">http://lab.db.ripe.net/whoi[…]nic&query-string=as3333</a><br />The given grs source id: 'ripe' is not valid<br /><br />And then: Why is there a distinction between lookup and grs-lookup as well as between search and grs-search? <a href="http://lab.db.ripe.net/whois/grs-lookup/apnic/aut-num/AS4808" rel="nofollow">http://lab.db.ripe.net/[…]/AS4808</a> and <a href="http://lab.db.ripe.net/whois/lookup/apnic/aut-num/AS4808" rel="nofollow">http://lab.db.ripe.net/whois/lookup/apnic/aut-num/AS4808</a> seem to be (almost) identical.
Anonymous •
Thank you Robert...<br /><br />The answer to the first question is that we are working on the GRS version of the RIPE Database; we temporarily removed it to release a stable version. The new release will also fix some minor bugs in the grs-search.<br /><br />The choice of having separate end-points is purely a design choice. GRS is a different service because it doesn’t returns live data instead it refreshes every night. But the main difference is that during the refresh process the data is transformed and sanitized in order to adapt the original data to RIPE RPSL format. It also removes any private data and discards objects that can’t be mapped to any object in the RIPE Database data model. Given these limitations, GRS will give a view of operational data from all the Registries.<br /><br />On the other hand lookup and search are "proxy services". They stream live data directly from the authoritative Internet Resource Registry. The search service supports also multiple Registries in the same query. These services however are subject to access control policies. <br /><br />For example a search request with “source=apnic&source=ripe” will run the query on the APNIC and RIPE Whois Servers, process and parse their live responses, remove duplicate objects and finally return a single XML or JSON response.<br /><br />For this reason we decided to have different service end-points: lookup and search to query live proxy data that is available from registries that have same RIPE data model (AfriNIC and APNIC) and grs-lookup and grs-search to query snapshot data produced by our Global Resource System. <br /><br />This way separate cohesive interfaces will be easier to test and to document than one interface whose parameters can have different semantics and whose responses can have different natures.<br /><br />I hope this clarifies matters...<br />
Hide replies
Anonymous •
Hi Paul, thanks for the answer!<br />From my side of the database the different endpoints right now do have the same semantics. When working with the REST API, this now leads to either code duplication for grs/live or to treating a source as a tuple (grs_flag, source). Also the /sources and /templates endpoints seem to equally apply to both groups of sources (grs and live). I guess calling the new grs sources ripe-grs, arin-grs, radb-grs and using the same endpoints would make coding a bit simpler. But I can see that making objects unique over a multi-source query will be much harder as soon as grs and live sources appear in the same query...<br /><br />Finally a sidenote. RADB seems to be not fully working.<br />e.g. <a href="http://lab.db.ripe.net/whois/grs-lookup/radb/as-set/AS-SWITCH" rel="nofollow">http://lab.db.ripe.net/[…]/AS-SWITCH</a> fails.
Hide replies
Anonymous •
Dear Robert,<br />what the services share are the response schema (that is the same for all our query services) and the URL template.<br />But they return different data and they are supposed to have different usages. <br />For specific requirements it may still make sense to merge responses from different services and that is easy to be done client side unless someone helps us identify useful use cases that we will than implement and expose as use case services, similarly to what we have done for the abuse finder service for example.<br /><br />About code duplication... it should not depend on the interfaces that you are going to use, for example if some services share the same response schema you can create a reusable module that parses or bind responses into objects for all these services.<br />Similarly if two services have similar signature, like serch and grs-search, you can create a URL builder that will reuse the same<br />URL template and so on.<br />These patterns can also come out of the box if you use one of the many REST frameworks for your programming language of choice.<br />For example some of these frameworks will create a client by just annotating an object method with a URL template, the framework will do<br />all the rest, from the http request to binding XML response to objects, mapping exceptions, etc.<br /><br />The /whois/templates/<object_type> service end-point doesn't include any source parameter because it's independent from sources, the templates are the same for all the registries that use RIPE RPSL and for all mirrored sources.<br /><br />And as you suggest we will extend the sources service in a next release, maybe we will return two groups of source, live and GRS.<br /><br />Finally about AS-SWITCH. That as-set exists in the RADB registry but as you can verify with:<br />whois -h whois.radb.net AS-SWITCH<br />it has source:RIPE in its attributes, it is a RIPE object that for some reason as been duplicated in RADB database. <br />During the mirroring process one of the many things we do is discard the objects that have a source different from the registry that is being mirrored. <br />In practice you will find that object in the RIPE GRS mirror when we will release it.<br /><br />Regards,<br />--<br />Benedetto<br />Software Engineer<br />RIPE NCC Database Group
Hide replies
Anonymous •
Hi Benedetto, thanks for elaborating on my points!