The RIPE Database contains a lot of useful information about who to contact in case of network abuse or spam. But sometimes it can be difficult to find the right abuse handler information for a network resource in the RIPE Database. We developed an Abuse Finder tool that will make it easier.
The RIPE Database contains a lot of useful information about who to contact in case of network abuse or spam. However, this information is scattered over various RPSL objects and is not consistent in coverage or quality.
In addition to that, there are several ways to get to the information and it isn’t at all straightforward. In other words: it’s hard to find the right abuse handler information for a network resource in the RIPE Database. There isn’t a simple database query one can construct that will return such information.
Until now…
While working on the new RIPE Database query API, we came up with the idea of “use case queries”. A “use case query” will return a concise answer to a question that would normally involve doing some manual data mining in the RIPE Database.
The “Abuse Finder” is the first of such data mining tools; the data mining is now done for you in the background.
How does the Abuse Finder tool work?
You only need to supply a network resource ID and press the search button. The tool will apply some business rules to assemble a list of abuse contact details. One of the benefits of this approach is that the tool doesn’t need to present any personal data and is therefore exempt from query blocking rules.
The tool will return the following data:
- A link to an IRT object if available
- A list of abuse mailboxes
- A list of links to objects with abuse related remarks
How does the tool harvest its results?
To adapt a popular saying: 'A good picture speaks a thousand words lines of code':
The tool will remove any duplicate results. It will also return links to objects that have any of the following keywords listed in the remarks field:
- Abuse
- Spam
- Complaint
- Trouble
- Problem
Because remark lines are free text, we felt that they required some (human) inspection to assess if the resulting remarks contained valid abuse handler data; therefore the tool just provides pointers to these objects.
Point your script at the Abuse Finder tool!
Anyone who has written scripts to access the RIPE Database knows that there are some rules to consider - especially for scripts that frequently query the database:
- Excessive query behaviour can cause the tool to time out. This is by design to prevent DDoS attacks.
- Queries may include personal data. There is a limit to the amount of personal data the service returns to any client. Clients may get blocked if they exceed the limits.
- Searches are very liberal with the amount of data it returns. Lots of data may need to be filtered out to get to the proper results.
- RPSL is a tricky format to parse well by code.
The “Abuse Finder” RESTful web service has addressed all of these issues by:
- Only returning relevant data in any one response.
- Not returning any personal data, so there is no chance of getting blocked.
- Responding with an easy to parse XML or JSON formatted response.
- In case the database structure were to change, the response by this tool may not have to change.
The API documentation will be updated soon to reflect this.
What’s next?
This time we are not only looking for feedback about this “Abuse Finder” tool. We are also interested to hear if there are other “Use Case” searches that you would find useful.
If the proposal gets enough thumbs up we may implement it as a proper service…
Please see also initial description of the RIPE Database API , the RIPE Database Query Including Search and a short RIPE Database Query API Update .
Comments 10
Comments are disabled on articles published more than a year ago. If you'd like to inform us of any issues, please reach out to us via the contact form here.
Anonymous •
<div class="content legacycomment"> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> Hello everybody, </span> </p> <p> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> I love this. I really do. We had such a hard time parsing whois output and creating our own </span> </p> <p> <span style="font-family: Arial;"> Abuse Contact Database ( <a href="http://abusix.org/services/abuse-contact-db"> http://abusix.org/services/abuse-contact-db </a> ) </span> </p> <p> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> As soon as your service is online I will start to retire our service, for RIPE region. </span> </p> <p> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> But I have some suggestions and ideas that could make sense. </span> </p> <p> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> Please have a look at the following 2 policy proposals: </span> </p> <p> </p> <p> <a> <span style="font-size: larger;"> </span> </a> </p> <p> </p> <p> <a href="http://www.apnic.net/policy/proposals/prop-079"> http://www.apnic.net/policy/proposals/prop-079 </a> </p> <p> </p> <p> <a> <span style="font-size: larger;"> </span> </a> </p> <p> </p> <p> <a href="http://www.afrinic.net/docs/policies/AFPUB-2010-GEN-002.htm"> http://www.afrinic.net/docs/policies/AFPUB-2010-GEN-002.htm </a> </p> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> The APNIC was accepted and within AfriNIC the feedback was good enough to say that it will </span> </p> <p> <span style="font-family: Arial;"> probably pass as well. After that we would like to proposae the same policy to RIPE as well. </span> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> So if there is a mandatory IRT Object with 2 email addresses (e-mail attribute for personal </span> </p> <p> <span style="font-family: Arial;"> contact and abuse-mailbox attribute for mass reports) does this give you any trouble. Or do </span> </p> <p> <span style="font-family: Arial;"> I have to be carefull while adjusting the policy papers to RIPE? </span> </p> <p> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> Another great feature would be a feedback API that gives users the opportunity to send you </span> </p> <p> <span style="font-family: Arial;"> feedback about accuracy of email addresses. For example, if an address is bouncing or not </span> </p> <p> <span style="font-family: Arial;"> existing, the API user could give you feedback and tell you that this email address is not working. </span> </p> <p> <span style="font-family: Arial;"> That way you could delete wrong addresses, contact owners, ... </span> </p> <p> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> Think about the fact that there is only an IRT Object with the above mentioned 2 addresses </span> </p> <p> <span style="font-family: Arial;"> and none of them is working, if you get feedback, you could stop serving these specific addresses </span> </p> <p> <span style="font-family: Arial;"> and give back the addresses from the range above. That way reporters could report their stuff to </span> </p> <p> <span style="font-family: Arial;"> an existing, working address and the abuse department above will put pressure on the network </span> </p> <p> <span style="font-family: Arial;"> owner with wrong abuse departments. </span> </p> <p> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> How about offering this as an DNS based service, where Users could query with 'host' or any </span> </p> <p> <span style="font-family: Arial;"> other easy to use tool. This would help a lot of people not to change their scripts to much. </span> </p> <p> <span style="font-family: Arial;"> Our Contact DB is serving several million queries per day. </span> </p> <p> </p> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> Other Use cases I could think of, would be </span> </p> <p> </p> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> IP(in) --> ASN(out) </span> </p> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> IP(in) --> CIDR(out) </span> </p> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> IP(in) --> CC(out) </span> </p> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> CIDR(in) --> smaller and bigger CIDRS(out) </span> </p> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> CIDR(in) --> CC(out) </span> </p> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> CIDR(in) --> ASN(out) </span> </p> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> ASN(in) --> all CIDRS(out) </span> </p> <p> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> I hope this was not to confusing, but I thought, lets write down what is in my head at the moment. </span> </p> <p> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> I think there is some other stuff up there, but I think this is enough for the moment. </span> </p> <p> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> Thanks again for this great idea and looking forward your feedback. </span> </p> <p> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> Tobias </span> </p> <p> </p> <p> <span style="font-family: Arial;"> <br /> </span> </p> <p> <span style="font-size: larger;"> </span> </p> <p> <span style="font-family: Arial;"> abusix.org </span> </p> <p> </p> </div>
Anonymous •
<div class="content legacycomment"> <p> </p> <p> Thank you Tobias for your feedback. The issue of documenting abuse handling data in the RIPE Database is on the agenda for both the Database and Anti-abuse Working Groups at RIPE 60 next week. We will look into your suggestions for additional address calculations. Some of this functionality may already be available in existing tools. </p> <p> </p> </div>
Hide replies
Anonymous •
<div class="content legacycomment"> <p> Unfortunately I can not attend RIPE 60 next week. But I would really like to hear what the DB and Abuse WG is thinking about our policy proposal to APNIC, AfriNIC and soon RIPE. </p> <p> </p> <p> Thanks, </p> <p> </p> <p> Tobias </p> </div>
Anonymous •
<div class="content legacycomment"> <p> </p> <p> On the Database Working Group and Anti-Abuse Working Group mailing lists we received some questions we answered about the Abuse Finder tool. Here they are in Q & A style: </p> <p> </p> <p> <strong> Q </strong> : <em> I personally would prefer scriptable stuff (anything that generates text output that may be scripted) Any hope for that ? </em> </p> <p> </p> <p> <strong> A </strong> : The "Abuse Finder" form actually uses an extension to the RIPE Database Query API, which is implemented in the form of a RESTful Web Service. </p> <p> The API allows you to write script against this search tool and responses can be received in either XML or JSON format for easy parsing. </p> <p> </p> <p> We will be updating the API documentation soon, but in the meantime you could try clicking the XML and JSON icons next to the response from the "Abuse Finder" form to see the original response from the web service. </p> <p> </p> <p> <strong> Q </strong> : <em> It would be helpful, if its noted in the output from wich object or field the abuse email address was extracted from (e.g. admin-c, tech-c, remarks, abuse-field also) so output could look like: </em> </p> <p> <em> <br type="_moz" /> <br /> </em> </p> <p class="rteindent1"> <em> admin-c: noc@tester.de </em> </p> <p class="rteindent1"> <em> abuse-email: abuse@tester.de </em> </p> <p class="rteindent1"> <em> <br type="_moz" /> <br /> </em> </p> <p> <em> so that one can decide, wich one is really relevant for their own needs. </em> </p> <p> </p> <p> <strong style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "> A </strong> : We did think about applying some weighting to the email addresses returned. However, the abuse handling data is not structured within the RIPE Database. There are many places the abuse handling email can be put. This depends on where the network administrator thinks is the most appropriate place. It could be in an IRT object. Or in the admin-c of the maintainer of the INETNUM object. Because it is very subjective, one address is no more valuable or important than another. In most cases you are unlikely to receive a long list of email address options. So the actual objects they came from is less important. </p> <p> </p> <p> <strong style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "> Q </strong> : <em> What are the exact [query] limits? Is there a way of raising the limits for special cases (we maintain our own spam blacklist and do send about 30.000 reports, where there are about 5.000 to 8.000 reports originate from the RIPE region) daily <br /> </em> </p> <p> </p> <p> <strong style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "> A </strong> : One of the main reasons for developing this tool is to reduce the need for users to query for personal data objects. In which case limits become less relevant. With the current data structure we cannot yet totally remove this need. Many abuse email addresses are contained in remarks, some of which are within personal data objects. We return links to the objects that contain remarks that may hold such an email address. </p> <p> </p> <p> If the tool does not return any "abuse-mailbox:" email addresses you will need to follow these links to the suggested objects with remarks. By pointing to those objects that have such remarks this tool avoids the need for users to directly query for all personal data objects referenced. </p> <p> </p> <p> <strong> Q </strong> : <em> How can the finder be tested via other protocols [then just a webpage]? </em> </p> <p> </p> <p style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "> <i> <b> <br /> </b> </i> </p> <div> <strong style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "> A </strong> : The back end web service can be accessed via any HTTP client library. The response is available in XML or JSON. Please refer to the <a href="http://labs.ripe.net/content/ripe-database-api-documentation"> API Documentation </a> for further details. <p> </p> </div> <p> </p> </div>
Hide replies
Anonymous •
<div class="content legacycomment"> <p> > A: We did think about applying some weighting to the email addresses returned. However, the abuse handling data is not structured within the RIPE Database. There are many <br /> > places the abuse handling email can be put. This depends on where the network administrator thinks is the most appropriate place. It could be in an IRT object. Or in the <br /> > admin-c of the maintainer of the INETNUM object. Because it is very subjective, one address is no more valuable or important than another. In most cases you are unlikely <br /> > to receive a long list of email address options. So the actual objects they came from is less important. </p> <p> </p> <p> That is one of the reasons for our policy proposal. Would it make sense to start this proposal pretty soon and wait with the abuse Finder until a decission is made? </p> <p> </p> <p> Thanks, </p> <p> Tobias </p> </div>
Anonymous •
<div class="content legacycomment"> <p> I agree with Tobias that some hints about how the e-mail addresses were selected </p> <p> is essential for this to be really useful. I see that it is difficult; however overcoming </p> <p> that difficulty would make the tool very much more useful. </p> <p> Providing black boxes that return e-mail addresses is a dangerous thing to do, because </p> <p> before you know it, tools and ignorant users alike will just use those addresses. </p> <p> Tagging them, or ordering them by relevance, may help here. </p> <p> </p> <p> I also note that the prototype has a tendency to list abuse@ripe.net more often than it should. </p> <p> Examples I came across: RFC1918 address space and 88.198.84.162. Some improvement </p> <p> of the heuristics is needed before abuse@ripe.net gets flooded with even more mail. </p> <p> </p> <p> One suggestion: Run the tool on all allocations and assignments made by the RIPE NCC. </p> <p> Then do some stats on all the returned e-mail addresses. This may provide some insight </p> <p> into where the heuristics can be improved. </p> <p> </p> <p> I agree that better definitions of where to put abuse contact information is needed, </p> <p> however that shold not prevent us from improving tools like this that use existing </p> <p> information and heuristics. As more structured abuse information becomes available </p> <p> it will only make these tools stronger. </p> <p> </p> <p> Daniel </p> </div>
Anonymous •
<div class="content legacycomment"> <p> </p> <p> On the Anti Abuse Mailing list we received a question if we could give an example of a simple query... </p> <p> </p> <p> Whenever a response is given by any of the Client Forms supplied with the API articles, when clicking on the following icon: </p> <p> </p> <p> <meta charset="utf-8" /> </p> <p> </p> <p> a new window is opened with the original query (HTTP Get) and results in XML in this case. You will also find this option in the <a target="_blank" href="http://lab.db.ripe.net/portal/abuse-finder.htm"> Abuse Finder </a> form. </p> <p> </p> <p> <meta charset="utf-8" /> For a more detailed description of the service please have a look at the <a href="http://labs.ripe.net/content/ripe-database-api-documentation"> API documentation </a> . </p> <p> </p> <p> </p> </div>
Anonymous •
<div class="content legacycomment"> <blockquote> <p> </p> <p> <span style="font-family: Arial;"> I am trying to find the bottom of the pit that opens with this suggestion: </span> </p> <p> </p> <p> <em> <span style="font-family: Arial;"> "Another great feature would be a feedback API that gives users the opportunity to send you </span> </em> <em> <span style="font-family: Arial;"> feedback about accuracy of email addresses </span> " </em> </p> <blockquote> <p> <sup> ° </sup> </p> </blockquote> <blockquote> <p> </p> </blockquote> </blockquote> <p> First of all, I think the RIPE NCC is not managing the registered eMail addresses. That's the responsibility of the maintainer for a particular entry. So if we collect such information it should rather be forwarded, or made available, to the responsible party. </p> <p> </p> <p> The other, and imho more fundamental aspect here is: are resource holders really required to receive or accept electronic mail from everyone and her dog? </p> <p> </p> <p> My personal opinion is: <strong> no </strong> . </p> <p> </p> <p> Not having email configured at all, not accepting email from some necks of the woods, and such, seems to be a perfectly valid decision to me. YMMV. </p> <p> </p> <p> All the best, </p> <p> Wilfried </p> </div>
martin •
http://lab.db.ripe.net/portal/abuse-finder.htm does not work... have no A/AAAA-Record.
Hide replies
Kaveh Ranjbar •
Hello, Thanks for spotting the issue. The link in the document was for the prototype version, at RIPE 62 we moved the service to production and made it available at https://apps.db.ripe.net/search/abuse-finder.html I have updated the document with new link. Kind Regards, Kaveh. --- Kaveh Ranjbar Database Group Manager, RIPE NCC