RIPE TTM User Survey Results

The RIPE NCC has provided the RIPE Test Traffic Measurement (TTM) service since 1997 [ ripe-158: original note for the service ]. There are 94 TTM boxes hosted by 69 host organisations. In September and October 2011, we conducted a survey among all TTM hosts to make an evolution plan that will help us decide the future of the TTM service.

Since we did not engage with the TTM hosts for quite some time and the development of the service itself was frozen, getting back in touch took some effort. In the end, the response was not bad at all: out of 69 organisations, we received 39 unique answers, covering 49 boxes.

The Results

For a first impression, we presented a number of statements and asked the TTM users whether they agreed with them or disagreed on a six point scale from full disagreement to full agreement.  We consider respondents to generally in agreement when the majority choose either 'mostly agree' or 'fully agree'. We consider respondents to be generally in disagreement when they choose the two options at the other end of the spectrum.

The first group of questions were about the general use and benefits of TTM (click to enlarge):

Figure 1: Usage and Benefits of TTM - Survey Results

Users consider the independence of the RIPE NCC important, which is what our experience suggests. It is more surprising to us that users still feel TTM to be useful to them and their organisation. The time server provided by the boxes is also popular. The answers to the remaining statements varied, and the usefulness of TTM for daily work and strategic planning is no longer universally recognised.

Then we wanted to know how users felt about the future of TTM:

Figure 2: Future  of TTM - Survey Results

We were quite surprised to find that almost all respondents expect to continue using TTM in 2012. We were less surprised to learn that respondents agree that TTM needs to evolve and that they would like the RIPE NCC to present plans for that evolution. Most respondents also consider it useful to investigate a possible integration of RIPE TTM with RIPE Atlas.

Finally, we were interested to see if respondents were willing to engage and help us plan the evolution of TTM. More than 80% of the respondents said that they would likely or definitely answer another such survey.

The Comments

In the survey, we made space for "free text" expression of experiences, opinions and wishes. Here is the summary of those comments.

The benefits that most people list are:

• Using TTM boxes as an independent clock source / NTP server (this got mentioned 8 times!)
• Monitoring SLA towards customers from an independent data source: this was expressed in multiple ways, such as:
• proving quality of their services to existing or new customers
• reacting to customer complaints, or
• proving that “the Internet is working” ;-)
• Troubleshooting: debugging networking incidents; monitoring reachability and performance; strategic planning

Several organisations also mentioned that the data is used for academic research, or that they are happy to host the box which provides information for others, although they do not have operational use of the measurements themselves. On the shortcomings side, the opinions were grouped into:

• Big lag in data access: data is available one day later
• User interface needs improvement
• There were some problems with GPS antennas reported, and
• Data representation needs more work

When we asked hosts to suggest improvements for TTM in the future, the main ones were about solving problems of the already listed shortcomings:

• Hosts are asking for the real-time data, which can be then used for alarms “ttm needs to be faster than tickets from our clients in case of connectivity problems”
• Alarms need to be configurable: hosts need to be able to set up own thresholds and parameters
• There is a need for another format of data access / data representation: either API, or DB
• Multiple answers suggested integration between TTM data and other monitoring and/or data analysis tool

Additional suggestions were:

• To have more TTM boxes deployed, or activated, because the benefits are only present when there is a rich network to use as a reachability reference
• Integration between Atlas and TTM

Some hosts of deactivated boxes also responded, giving us insight into why they have chosen to discontinue the service: mostly because they found the maintenance costs high. Due to the economic situation in their company or country, they were forced to cut any non-essential operational costs.

Conclusions

As the TTM service is  more than 10 years old, it is quite “ripe” for  improvement (pun intended). The majority of the current users are still happy with the service but sees the need for TTM to evolve. Integration with RIPE Atlas is considered positively and the RIPE NCC is asked to present an evolution plan for TTM.

We will now work on this plan together with the current users who expressed their willingness and with the RIPE Measurements, Analysis and Tools (MAT) Working Group.

You are welcome to leave a comment, discuss in the MAT WG or to send a private comment or question to ttm@ripe.net .