Stephen Strowes

Here to Stay: RIPE Atlas Daily Archives

Stephen Strowes
4

A few months ago we started offering daily archives of public RIPE Atlas measurements. We're happy to announce that this service is here to stay!


Background

RIPE Atlas measurement data is always available via the API, but many distinct measurements run every single day and aggregating the results manually can be a pain. We collect a rich set of measurement data and, as good citizens, we don't want to generate more measurement traffic than is necessary. It's important that we make this data as easy to retrieve as possible.

And you all want a ton of data, right? Right.

Last year, we prototyped a service to expose this data for around one month from the day of collection. The feedback has all been positive, so we've worked on making it more permanent.

Minor Updates

In making this more permanent, we're also making some minor modifications that you should take into account.

First, the new location for the data:

https://data-store.ripe.net/datasets/atlas-daily-dumps

Second, the directory structure and file naming has changed. We're now bundling all measurements of a particular type into one; that means that each bzip archive will, for example, contain all ping measurements in the given time window. Previously, those measurements were spread across multiple archives for IPv6, IPv4, built-in, and user-defined measurements.

All of those subsets still exist within the data, but if you specifically want those subsets you should do the following:

  • To split out IPv6 from IPv4, most of the results in the files are tagged with an 'af' field, except in certain error conditions where a measurement was attempted but did not generate any results.
  • Built-in measurements always have a msm_id value less than 1,000,000.

The other difference reflected in the filenames is that we are now generating archives per hour rather than per day. This makes the file sizes a little more manageable for download, and may help with sampling the data.

Finally, the archive will continue to hold data for about one month from the date of collection.

Please update your scripts; the old archive at https://ftp.ripe.net/ripe/atlas will be removed on 15 March 2018.

Be aware that release of this data falls under the regular RIPE NCC Terms of Service. Finally, if you spot any glitches or have any problems using the service, please contact us at atlas@ripe.net!

Tags:
4

You may also like

View more

About the author

Stephen is a principal engineer in the Office of the CTO at Fastly, where he spends his time trying to figure out what the Internet is. He obtained his PhD in Internet routing scalability from the University of Glasgow in 2012.

Comments 4