RIPE Atlas Update: HTTP Measurements, CLI Tools, DomainMON and More

In addition to the two important new RIPE Atlas features described in recent RIPE Labs articles, Time Travel and LatencyMON , we've been busy working on a long list of enhancements, new features and other news:

1. Sharing my probe with other individuals
2. HTTP measurements towards RIPE Atlas anchors
3. More prominent exposure of built-in measurements
4. Command-Iine interface (CLI) tools
5. Blacklisting and whitelisting targets
6. Healing (anchoring) measurements
7. Heads-up: v2 APIs are coming
8. DomainMON
9. RIPE Atlas outreach and e-learning

Sharing my probe with other individuals

Sharing administrative access of your RIPE Atlas probe with other members of your LIR has been possible for a long time, as long as you are a RIPE NCC member. Now you can share your probe's administration with individuals, regardless of whether they are members of your LIR or not. Keep in mind that this gives full control to the probe, including removing the original host's administrative access or transferring the probe to a new host.

To take advantage of this feature, just go to the general tab of the probe's status page .

HTTP measurements towards anchors

There have been a number of discussions at RIPE Meetings and on various mailing lists (such as the MAT-WG mailing list or the RIPE Atlas public mailing list ) about the support for general HTTP measurements in RIPE Atlas (see the earlier RIPE Atlas article HTTP Measurements with RIPE Atlas ). Unfortunately, there are risks of allowing any user to measure any web server with RIPE Atlas regarding both the probe hosts and the target servers. In order to mitigate these risks, we decided to restrict the targets to RIPE Atlas anchors . These are willing targets, and they provide known and short responses. If you would like to measure HTTP towards your own network, you can apply for an RIPE Atlas anchor .

We also plan to add HTTP measurements by default into the anchoring measurements, which are run with several hundred probes per anchor – stay tuned!

More control over probes' spread within the measurement interval

One of the very first features of user-defined measurements was to let users define the frequency of their measurements. However, users didn't have influence over how probes were distributed inside this interval; this was fully controlled by the scheduler.

The new spread control, available via the API and the web interface, now allows users to tailor this to their specific needs. In addition to specifying the measurement frequency, users can also specify the duration of a time interval within which the probes will be linearly distributed.

For example, if you choose a measurement frequency of 3,600 seconds, and a spread of 3,600 seconds as well, then the system will try to randomise all probes within the full interval. The target will see a more or less steady packet inflow. If you choose instead to make the interval a small number, then the probes will congregate close to the frequency "ticks", causing higher peaks. In both cases, the probes will stick to the specified frequency and not move randomly within the allowed interval.

Figure 1: Interval control in action. The x-axis is the time of measurement and dots are individual results. The green dots correspond to a frequency of 1,800s and a spread of 30s, while red dots correspond to a frequency of 3,600s and a spread of 3,600s.

In order to avoid overloading the targets with bursts of measurements, the system enforces a sane limit on how high the peaks can be (i.e. maximum queries per second).

More prominent exposure of built-in measurements

The probes constantly execute built-in measurements , and have been since day one of RIPE Atlas. Getting the results of these measurements has also been possible and relative easy for a while, but we saw that there was still room for improvement.

You can now see the list of these built-in measurements, basic information about them, data downloads, and, in some cases, even visualisations in a new tab in the list of measurements.

Command-line interface (CLI) toolkit

We've received many requests for a command-line interface (CLI) for creating measurements and downloading results. We're pleased to announce that this toolkit is now available. It is written and maintained by the RIPE NCC, but it relies on contributions from the community. The code is open source and is available on GitHub . It already has many contributors, as you can see in Figure 2 below.

Figure 2: Number of contributors to the RIPE Atlas CLI toolkit on GitHub

Please also note the documentation available for the CLI toolkit.

Figure 3: Documentation interface for RIPE Atlas CLI

We invite you to try it out, give us your feedback and contribute patches yourself!

Blacklisting and whitelisting targets

Several users mentioned that they don't really mind how much measurement traffic is destined to their probes. These probes are usually in larger networks that are run by larger organisations and have anycast installations or use multiple sites to serve their users. In these cases, the system-enforced limits on how many measurements can run concurrently against the same target are mostly just in the way.

On the opposite side, some network operators prefer not to have their servers measured at all, or perhaps don't want to be measured except for a few specific addresses.

We expect more and more users to request being added either to a whitelist or blacklist as RIPE Atlas grows. If you would like to be added to this whitelist or blacklist, please let us know .

Healing (anchoring) measurements

Anchoring measurements are automatically executed against each RIPE Atlas anchor from several hundred probes, and include IPv4 and IPv6 ping and traceroute measurements. The point of these is to "anchor" the probes to  stable targets, but also to provide a steady stream of results for the anchors themselves. Over short time periods, these help illustrate connection or path problems; over longer time periods, they help illustrate the evolution of connectivity or path changes.

As you can imagine, over longer periods, some probes will stop participating in the network and, as a result, these anchoring measurements will slowly use fewer and fewer probes. We're in the process of swapping these "abandoned" probes with new ones to maintain the usefulness of these measurements.

The concept is also applicable to generic user-defined measurements, since long-running user-defined measurements can also suffer from degradation. We started working on mechanisms to at least recognise these situations, and perhaps even to automatically "heal"  user-defined measurements when certain conditions are met.

Heads-up: v2 APIs are coming

RIPE Atlas v1 APIs have been highly useful for years now for users who want to automate tasks, interact with our system from other systems, or get more information about probes or measurements.

However, over time technologies evolve and we need to respond to this if we don't want the system to become unmaintainable. In particular, due to version changes in our underlying framework, we have to abandon the module that powers the v1 API. In fact, this already happened under the hood: the v1 API right now is just a translation of the v2 API that we're already using internally. Eventually, we'll abandon this "life support", and only the v2 API will be available.

This is planned for next year, and we'll communicate the steps about this transition clearly and provide ample time for our users to switch to the new version.

Note that if you're using helper modules (such as RIPE Atlas Cousteau or even the RIPE Atlas toolset ), then this change could be entirely invisible to you, as these libraries abstract away the actual API calls. Therefore we encourage API users to check out these tools.

DomainMON - coming soon

DomainMON is an upcoming RIPE Atlas feature that many of our users have requested. It is a light-weight version of DNSMON for monitoring private domains from up to 50 probes using users' own RIPE Atlas credits. (DNSMON can only be used for monitoring generic and country code top-level domains (gTLDs and ccTLDs), and uses RIPE Atlas anchors exclusively as vantage points.)

The figure below shows an example of how to set up a DomainMON measurement for the domain belastingdienst.nl.

Figure 4: Setting up a DomainMON measurement

Below you can see the results of this measurement.
 

Figure 5: Results of DomainMON measurement to domain belastingdienst.nl

We are inviting testers to try out this new feature, and give us feedback. Please join the mailing list for beta-testers .

RIPE Atlas outreach and e-learning

We now provide a RIPE Atlas webinar for RIPE NCC members who want to learn more about RIPE Atlas and its capabilities. Check out the dates and sign up today! We'll also make a recording of one of the webinars available to everyone in the next few weeks.

There's a new RIPE Atlas Wikipedia entry - feel free to contribute if you have something to add.

The September issue of the Internet Protocol Journal is dedicated to RIPE Atlas (you can get the PDF here ) and goes into detail about its history, technical set-up, use cases and much more. Thanks to Ole Jacobsen for making this possible and thanks to all the contributors. 

We've been crowdsourcing RIPE Atlas documentation in multiple languages on GitHub and it is now available in Japanese, Spanish, Italian and French. Check it out, and thanks to everyone who contributed.

We are also proud to report that the RIPE Atlas network now contains more than 8,900 active probes in over 3,000 ASNs and more than 150 RIPE Atlas anchors .

Figure 6: Distribution of RIPE Atlas anchors

Please check out the collection of use cases & user stories on RIPE Labs to learn what others have done with this globally distributed network - and maybe get inspired about how you can use RIPE Atlas, too.

If you would like to talk to the RIPE Atlas developers or a RIPE Atlas ambassador, please consult the conference calenda r. And you can always reach us at atlas [at] ripe [dot] net.

You can stay up to date about RIPE Atlas developments by subscribing to the mailing list or take a look at the RIPE Atlas roadmap .