You are here: Home > Publications > RIPE Labs > Paul Palse > Find Abuse Handler Details Using the Abuse Finder

Find Abuse Handler Details Using the Abuse Finder

Paul Palse — Mar 2010
The RIPE Database contains a lot of useful information about who to contact in case of network abuse or spam. But sometimes it can be difficult to find the right abuse handler information for a network resource in the RIPE Database. We developed an Abuse Finder tool that will make it easier.

The RIPE Database contains a lot of useful information about who to contact in case of network abuse or spam. However, this information is scattered over various RPSL objects and is not consistent in coverage or quality.

In addition to that, there are several ways to get to the information and it isn’t at all straightforward. In other words: it’s hard to find the right abuse handler information for a network resource in the RIPE Database. There isn’t a simple database query one can construct that will return such information.

Until now…

While working on the new RIPE Database query API, we came up with the idea of “use case queries”. A “use case query” will return a concise answer to a question that would normally involve doing some manual data mining in the RIPE Database.

The “Abuse Finder” is the first of such data mining tools; the data mining is now done for you in the background.

 

Abuse Finder

 

https://apps.db.ripe.net/search/abuse-finder.html

How does the Abuse Finder tool work?

You only need to supply a network resource ID and press the search button. The tool will apply some business rules to assemble a list of abuse contact details. One of the benefits of this approach is that the tool doesn’t need to present any personal data and is therefore exempt from query blocking rules.

The tool will return the following data:

  • A link to an IRT object if available
  • A list of abuse mailboxes
  • A list of links to objects with abuse related remarks

How does the tool harvest its results?

To adapt a popular saying:  'A good picture speaks a thousand words lines of code':

 

Abuse Finder Flow

 

The tool will remove any duplicate results. It will also return links to objects that have any of the following keywords listed in the remarks field:

  • Abuse
  • Spam
  • Complaint
  • Trouble
  • Problem

Because remark lines are free text, we felt that they required some (human) inspection to assess if the resulting remarks contained valid abuse handler data; therefore the tool just provides pointers to these objects.

Point your script at the Abuse Finder tool!

Anyone who has written scripts to access the RIPE Database knows that there are some rules to consider - especially for scripts that frequently query the database:

  • Excessive query behaviour can cause the tool to time out. This is by design to prevent DDoS attacks.
  • Queries may include personal data. There is a limit to the amount of personal data the service returns to any client. Clients may get blocked if they exceed the limits.
  • Searches are very liberal with the amount of data it returns. Lots of data may need to be filtered out to get to the proper results.
  • RPSL is a tricky format to parse well by code.

The “Abuse Finder” RESTful web service has addressed all of these issues by:

  1. Only returning relevant data in any one response.
  2. Not returning any personal data, so there is no chance of getting blocked.
  3. Responding with an easy to parse XML or JSON formatted response.
  4. In case the database structure were to change, the response by this tool may not have to change.

The API documentation will be updated soon to reflect this.

What’s next?

This time we are not only looking for feedback about this “Abuse Finder” tool. We are also interested to hear if there are other “Use Case” searches that you would find useful.

If the proposal gets enough thumbs up we may implement it as a proper service…

Please see also initial description of the RIPE Database API , the RIPE Database Query Including Search and a short RIPE Database Query API Update .



10 Comments

Anonymous says:
27 Apr, 2010 09:49 AM

 

Hello everybody,

 

I love this. I really do. We had such a hard time parsing whois output and creating our own

Abuse Contact Database ( http://abusix.org/services/abuse-contact-db )

 

As soon as your service is online I will start to retire our service, for RIPE region.

 

But I have some suggestions and ideas that could make sense.

 

Please have a look at the following 2 policy proposals:

 

http://www.apnic.net/policy/proposals/prop-079

 

 

http://www.afrinic.net/docs/policies/AFPUB-2010-GEN-002.htm

 


The APNIC was accepted and within AfriNIC the feedback was good enough to say that it will

probably pass as well. After that we would like to proposae the same policy to RIPE as well.


So if there is a mandatory IRT Object with 2 email addresses (e-mail attribute for personal

contact and abuse-mailbox attribute for mass reports) does this give you any trouble. Or do

I have to be carefull while adjusting the policy papers to RIPE?


Another great feature would be a feedback API that gives users the opportunity to send you

feedback about accuracy of email addresses. For example, if an address is bouncing or not

existing, the API user could give you feedback and tell you that this email address is not working.

That way you could delete wrong addresses, contact owners, ...


Think about the fact that there is only an IRT Object with the above mentioned 2 addresses

and none of them is working, if you get feedback, you could stop serving these specific addresses

and give back the addresses from the range above. That way reporters could report their stuff to

an existing, working address and the abuse department above will put pressure on the network

owner with wrong abuse departments.


How about offering this as an DNS based service, where Users could query with 'host' or any

other easy to use tool. This would help a lot of people not to change their scripts to much.

Our Contact DB is serving several million queries per day.

 


Other Use cases I could think of, would be

 

IP(in) --> ASN(out)  

IP(in) --> CIDR(out)

IP(in) --> CC(out)

CIDR(in) --> smaller and bigger CIDRS(out)

CIDR(in) --> CC(out)

CIDR(in) --> ASN(out)

ASN(in) --> all CIDRS(out)


I hope this was not to confusing, but I thought, lets write down what is in my head at the moment.

I think there is some other stuff up there, but I think this is enough for the moment.


Thanks again for this great idea and looking forward your feedback.


Tobias


abusix.org

Anonymous says:
27 Apr, 2010 03:01 PM

 

Thank you Tobias for your feedback. The issue of documenting abuse handling data in the RIPE Database is on the agenda for both the Database and Anti-abuse Working Groups at RIPE 60 next week. We will look into your suggestions for additional address calculations. Some of this functionality may already be available in existing tools.

 

Anonymous says:
27 Apr, 2010 04:02 PM

Unfortunately I can not attend RIPE 60 next week. But I would really like to hear what the DB and Abuse WG is thinking about our policy proposal to APNIC, AfriNIC and soon RIPE.

 

Thanks,

 

Tobias

Anonymous says:
27 Apr, 2010 03:48 PM

 

On the Database Working Group and Anti-Abuse Working Group mailing lists we received some questions we answered about the Abuse Finder tool. Here they are in Q & A style:

 

Q I personally would prefer scriptable stuff (anything that generates text output that may be scripted) Any hope for that ?

 

A : The "Abuse Finder" form actually uses an extension to the RIPE Database Query API, which is implemented in the form of a RESTful Web Service.

The API allows you to write script against this search tool and responses can be received in either XML or JSON format for easy parsing.

 

We will be updating the API documentation soon, but in the meantime you could try clicking the XML and JSON icons next to the response from the "Abuse Finder" form to see the original response from the web service.

 

Q It would be helpful, if its noted in the output from wich object or field the abuse email address was extracted from (e.g. admin-c, tech-c, remarks, abuse-field also) so output could look like:



admin-c: noc@tester.de

abuse-email: abuse@tester.de



so that one can decide, wich one is really relevant for their own needs.

 

A : We did think about applying some weighting to the email addresses returned. However, the abuse handling data is not structured within the RIPE Database. There are many places the abuse handling email can be put. This depends on where the network administrator thinks is the most appropriate place. It could be in an IRT object. Or in the admin-c of the maintainer of the INETNUM object. Because it is very subjective, one address is no more valuable or important than another. In most cases you are unlikely to receive a long list of email address options. So the actual objects they came from is less important.

 

Q What are the exact [query] limits? Is there a way of raising the limits for special cases (we maintain our own spam blacklist and do send about 30.000 reports, where there are about 5.000 to 8.000 reports originate from the RIPE region) daily

 

A : One of the main reasons for developing this tool is to reduce the need for users to query for personal data objects. In which case limits become less relevant. With the current data structure we cannot yet totally remove this need. Many abuse email addresses are contained in remarks, some of which are within personal data objects. We return links to the objects that contain remarks that may hold such an email address.

 

If the tool does not return any "abuse-mailbox:" email addresses you will need to follow these links to the suggested objects with remarks. By pointing to those objects that have such remarks this tool avoids the need for users to directly query for all personal data objects referenced.

 

Q How can the finder be tested via other protocols [then just a webpage]?

 


A : The back end web service can be accessed via any HTTP client library. The response is available in XML or JSON. Please refer to the API Documentation for further details.

 

 

Anonymous says:
27 Apr, 2010 04:05 PM

> A: We did think about applying some weighting to the email addresses returned. However, the abuse handling data is not structured within the RIPE Database. There are many
> places the abuse handling email can be put. This depends on where the network administrator thinks is the most appropriate place. It could be in an IRT object. Or in the
> admin-c of the maintainer of the INETNUM object. Because it is very subjective, one address is no more valuable or important than another. In most cases you are unlikely
> to receive a long list of email address options. So the actual objects they came from is less important.

 

That is one of the reasons for our policy proposal. Would it make sense to start this proposal pretty soon and wait with the abuse Finder until a decission is made?

 

Thanks,

Tobias

Anonymous says:
30 Apr, 2010 09:11 AM

I agree with Tobias that some hints about how the e-mail addresses were selected

is essential for this to be really useful. I see that it is difficult; however overcoming

that difficulty would make the tool very much more useful.

Providing black boxes that return e-mail addresses is a dangerous thing to do, because

before you know it, tools and ignorant users alike will just use those addresses.

Tagging them, or ordering them by relevance, may help here.

 

I also note that the prototype has a tendency to list abuse@ripe.net more often than it should.

Examples I came across: RFC1918 address space and 88.198.84.162. Some improvement

of the heuristics is needed before abuse@ripe.net gets flooded with even more mail.

 

One suggestion: Run the tool on all allocations and assignments made by the RIPE NCC.

Then do some stats on all the returned e-mail addresses. This may provide some insight

into where the heuristics can be improved.

 

I agree that better definitions of where to put abuse contact information is needed,

however that shold not prevent us from improving tools like this that use existing

information and heuristics. As more structured abuse information becomes available

it will only make these tools stronger.

 

Daniel

Anonymous says:
27 Apr, 2010 04:10 PM

 

On the Anti Abuse Mailing list we received a question if we could give an example of a simple query...

 

Whenever a response is given by any of the Client Forms supplied with the API articles, when clicking on the following icon:

 

 

 

a new window is opened with the original query (HTTP Get) and results in XML in this case.  You will also find this option in the Abuse Finder  form.

 

For a more detailed description of the service please have a look at the API documentation .

 

 

Anonymous says:
01 Jun, 2010 12:54 PM

 

I am trying to find the bottom of the pit that opens with this suggestion:

 

"Another great feature would be a feedback API that gives users the opportunity to send you feedback about accuracy of email addresses "

°

First of all, I think the RIPE NCC is not managing the registered eMail addresses. That's the responsibility of the maintainer for a particular entry. So if we collect such information it should rather be forwarded, or made available, to the responsible party.

 

The other, and imho more fundamental aspect here is: are resource holders really required to receive or accept electronic mail from everyone and her dog?

 

My personal opinion is: no .

 

Not having email configured at all, not accepting email from some necks of the woods, and such, seems to be a perfectly valid decision to me. YMMV.

 

All the best,

Wilfried

martin says:
25 Feb, 2012 04:51 PM
http://lab.db.ripe.net/portal/abuse-finder.htm does not work... have no A/AAAA-Record.
Kaveh Ranjbar says:
28 Feb, 2012 01:39 PM
Hello,

Thanks for spotting the issue. The link in the document was for the prototype version, at RIPE 62 we moved the service to production and made it available at https://apps.db.ripe.net/search/abuse-finder.html

I have updated the document with new link.

Kind Regards,
Kaveh.

---
Kaveh Ranjbar
Database Group Manager, RIPE NCC
Add comment

You can add a comment by filling out the form below. Comments are moderated so they won't appear immediately. If you have a RIPE NCC Access account, we would like you to log in.