Non-Public Measurements in RIPE Atlas

First of all, some history behind this topic. We recognise that there have been discussions about the topic before, including  a previous RIPE Labs article , which triggered a  discussion on our mailing list . At that time, the conclusion was that there should be no changes in policy, so we kept to the "status quo". However, since that was almost three years ago, it may be worthwhile to revisit and re-evaluate the topic in light of the current situation.

We define a non-public measurement as one that behaves exactly like any other measurement from the specification and execution point of view, except that it's not shown in the list of known measurements, and the results delivered are only available to the user who specified the measurement. A minor exception to this is that the hosts of probes that are involved in these measurements can also know about the existence of such measurements, because they have a right to know what their probes actually do.

A few statistics about the proportion of non-public measurements, as of 12 October 2016:

• Since the start of RIPE Atlas: 5.8%, from 1,758 different users
• Since 1 January 2016: 2.9%, from 33 different users
• Since 1 August 2016: 0.03%, from 5 different users

A different perspective on the same topic: in the past 12 months, 0.3% of our active users used non-public measurements at least once (the distribution is far from balanced). These numbers illustrate that this option is sparsely used though they are "paid by all", and was one of the top reasons that made us want to re-evaluate the need for the feature.

There are two underlying reasons for these numbers. One is that for a while now the user interface does not facilitate the creation of non-public measurements, whereas the API still allows it. The idea behind this is to encourage public measurements as much as possible while still allowing non-public ones for people who "really need" them. The other reason is that measurement campaigns are unpredictable; every now and then we see a batch of non-public measurements scheduled, which skews statistics; it's entirely possible that tomorrow we'll have 90% of new measurements specified as non-public.

It's clear that some users prefer to have the ability to execute non-public measurements. There are various reasons for this, some of which were already mentioned in previous discussions. For example, some operators would want to test reachability between two endpoints, and while they can use RIPE Atlas to measure, they consider the existence of these measurements and their results non-shareable. Another reason could be to test an upcoming network or server before it is officially available, since publishing such measurements would be counter-productive for the operator.

The question the RIPE Atlas team would like to ask our community is: should we continue supporting non-public measurements?

As Daniel Karrenberg hinted about this in his email , support for non-public measurements indeed complicates the RIPE Atlas codebase and makes some processes or functions more complex than they otherwise could be. It's also true that these complexities already exist in our code, so the long term issue is not the creation of these features, but their ongoing maintenance (and perhaps extension), as well as the way they affect upcoming functions. One already mentioned example here is the streaming service, where for each result we have to decide if the service is available to the current listeners or not. Another example would be if we wanted to have a partner help us in serving all our data: such a partner would either have to maintain the same set of access rules, or we would have to build-in mechanisms that weed out the non-public data before they are sent on.

The main possibilities going forward are the following:

1. Keep everything as it is now: non-public measurements are possible and their data is not generally available.
2. Keep non-public measurements, but introduce a grace period (say, a month) after which the measurement specification and results automatically become public. This does not simplify the service; in fact it makes it more complex to run as each access will have to take into account the time dimension, in addition to everything else it does now.
3. Remove the feature to schedule non-public measurements.

Option 2 provides the opportunity for users to continue making non-public measurements on the understanding that these measurements will be made public after a certain period. As indicated, however, whilst this might prove useful to those who wish to make measurements that are not to be made immediately available to others, choosing this option would increase the complexity of the service.

Also, it should be noted that going ahead with options 2 or 3 raises questions as to what is to be done with non-public measurements made in the past. We recognise that, up until now, non-public measurements have been made on the understanding that they would not become publicly available at a later date. So, in order to ensure that the relevant measurements will not go public in the future, we have two further options:

1. All non-public measurements made in the past remain non-public for good.
2. All non-public measurements made in the past are deleted, but only at the end of a grace period during which users have the opportunity to copy any prior non-public measurements they wish to keep.

Of the three initial options, the preference of the RIPE NCC would be to go with option 3 and remove the feature to schedule non-public announcements. This would remove the extra complexity introduced by non-public measurements in both the code base and the maintenance of measurements. But we are curious to hear from you. Please leave a comment below and/or fill in the two short surveys on top of the article.

Note that selecting some of these options may trigger a need to revisit the RIPE Atlas Service Terms and Conditions.