You are here: Home > Publications > RIPE Labs > Paul Palse > Free Text Search in the RIPE Database

Free Text Search in the RIPE Database

Paul Palse — 10 Nov 2010
We have implemented an improved free text search for the RIPE Database based on Solr. Find the details and a pointer to this new search function in the article below.

Introduction

We currently offer the possibility to do a free text search within the RIPE Database from our website www.ripe.net. All we can say about the current search method is that it "somewhat works". The search functionality is not quite what we are used to when using web searches.

A new site, a new search engine

The new RIPE NCC website, which will be launched early next year, will use a different portal framework for which a new search engine has been implemented. The underlying free text indexing engine for the new website, Solr , is a search framework from the Apache Lucene project which can easily be extended and has a large community backing.

The current free text search on www.ripe.net uses a special set of database text dump files that form the search index. The new Solr indexer uses direct database access to produce the index, which eliminates the extra step of dumping special files.

The new search complies with the RIPE Database Terms and Conditions and therefore doesn't index any personal data. PERSON and ROLE objects are not indexed at all. However when opening the details of the search results, it is easy to navigate to the related PERSON and ROLE object details.

Improvements and new features

Aside from being faster and producing more accurate results, the new search offers some new useful features:

Filter by object type

Any kind of search frequently returns a lot of hits within different object types. We present all results, plus the option to filter results by object type.

Searching for multiple keyword

It is possible to use "Logical Operators" in the search by simply specifying if the search should produce results where results contain

  • ALL matching keywords,
  • ANY matching keyword or
  • only EXACT matches

Search within specific objects and/or attributes

It is possible to restrict the search to only be within certain object types or even within certain attributes within certain objects, using the advanced search. This is a very powerful search feature.

The good news, its available now!

We have published the new free text search on the RIPE Database Lab environment already. This way you can test the service and provide us with some feedback. When the new www.ripe.net site goes live we will integrate this search feature into the new website.

RIPE DB New Search Function

Have a look at:

http://lab.db.ripe.net/portal/free-text/search.htm

As usual, please let us know what you think.

 

 

11 Comments

Anonymous says:
10 Nov, 2010 12:49 PM

This free text search is great - well done!

It doesn't seem to extend to poems in the RIPE database, so I find the -i option on the command line very useful still.

Regards.

Mike Norris
Anonymous says:
10 Nov, 2010 02:22 PM
Dear Mike

We will investigate including poems and poetic-form in the next iteration of the service. We excluded them on the first iteration as poems often contain many words, far more than a typical database object. We did not want them to skew the search results. We could add an 'option' to include poems in the search results.

Regards
Denis Walker
Business Analyst
RIPE NCC Database Group
K.Rastas says:
17 Nov, 2011 02:16 PM
Hi, the link seems to be broken now, 17.11.2011 ? HTTP 404 error.

Regards,
Kristian Rastas
Bogdan Dumitrescu says:
17 Nov, 2011 02:48 PM
Dear Kristian,


Thank you for your comment. The full-text search has been promoted to a production service and is now available here:

http://apps.db.ripe.net/search/full-text.html


Best regards,

Bogdan Dumitrescu
Software Engineer
RIPE NCC Database Group
Daniel R says:
22 May, 2012 04:25 PM
I'm using your API to do searches, but cannot find a way to search the full text index with it. How do I use the API to search the full text index?
Kaveh Ranjbar says:
22 May, 2012 04:48 PM
Our API only supports queries and updates to main RIPE Database Service and full text search is no supported via the API.

Kind Regards,
Kaveh.

---
Kaveh Ranjbar,
RIPE Database Group Manager
Niall Donaghy says:
07 Jan, 2015 04:47 PM
Has there been or are there plans to update the API to support this?

Kind regards,
Niall
Tim Bruijnzeels says:
12 Jan, 2015 03:10 PM
Hi Niall,

At this time there are no plans to extend the API to support this.

We believe that the primary use case for this is for ad-hoc searches done by humans. But, please let us know if you see a good use case for this functionality in 3rd party tooling or scripts.


Kind regards

Tim Bruijnzeels
Assistent Manager Software Engineering
RIPE NCC
Ross says:
14 Jul, 2015 09:25 AM
Hi

Is it possible to add an option to export data from the full text search (json xml or just text).... and if it is not possible is there any chance you could make the feature available?

Many thanks

Ross
Tim Bruijnzeels says:
15 Jul, 2015 03:59 PM
We are currently focussed on improving the usability of the RIPE Database update UI because we see a lot of issues regarding authentication, and the management of contact information, route objects and reverse DNS. Long term we also want to improve the query interface, but for the time being that has a lower priority.

Kind regards

Tim Bruijnzeels
Assistent Manager Software Engineering
RIPE NCC
Ricky Beam says:
22 Aug, 2015 02:41 AM
> good use case for this functionality

SPAM hunting. Spammer do everything possible to make themselves hard to find. (i.e. they use different handles for anything that's indexed / searchable) The "descr" field is not searchable via any API at RIPE. The full-text search interface is the only way (short of mirroring the entire whois db), but that output is formatted for humans and thus a pain in the ass to process.

If 'http://rest.db.ripe.net/sea[…]=route&query-string=...' worked, it wouldn't take HOURS to track a spammer.
Add comment

You can add a comment by filling out the form below. Comments are moderated so they won't appear immediately. If you have a RIPE NCC Access account, we would like you to log in.