Quantifying Quality - A Community Engagement Quarterly Report

When I was entrusted with the responsibility of becoming the RIPE NCC’s Chief Community Officer, I gave a presentation - first to staff and then at various community events - in which I laid out what I considered the top three priorities in our community engagement at the RIPE NCC. Keeping things high level, the focus was:

• to improve and institutionalise our understanding of community needs;
• to optimise the workflows and how we meet those needs;
• to build ecosystems around the services we provide

The logic tying these points together is simple. By actively seeking input from the community, we gain valuable insights into their needs. This deeper understanding enables us to tailor our offerings, providing knowledge and services that are more relevant, valuable, and aligned with their expectations. As a result, excitement and appreciation builds within the community, as they recognise the value of what we bring. This leads to increased usage of our services, active participation in discussions, sharing feedback, offering suggestions for improvements, and potentially becoming ambassadors for our offerings to others.

This report aims to provide specific examples and outcomes resulting from our feedback-driven approach.

Activities and Outcomes

As mentioned in my previous report- pushing the needle - the RIPE NCC has adopted the Objective and Key Result (OKR) framework to focus on the qualitative outcomes of our activities and help us determine if they’re getting us closer to achieving our desired impact.

Diversity and Inclusion

In our role as secretariat, and as groundwork for keeping up a healthy discourse with the community, many of our activities are aimed at supporting the RIPE community’s efforts in remaining an open and inclusive environment for dialogue and feedback.

We know that events the RIPE NCC organise serve as an important entry point to the community for many. While RIPE meetings are primarily held in Europe, the RIPE NCC regional meetings aim to overcome barriers to engagement, such as time and financial constraints associated with attending meetings in Europe.

For years the RIPE NCC has been supporting the organisation of three regional meetings: the Middle East Network Operator Group and Peering Forum (MENOG), the RIPE NCC South East European Meeting (SEE), and the Eurasia Network Operators Group (ENOG).

In recent years, ENOG has been suffering from a gradual decline in regional engagement and ultimately came to a halt due to the war in Ukraine. This situation presented an opportunity for the emergence of the Central Asian Peering and Interconnection Forum (CAPIF), which was designed specifically to address the peering and interconnection needs of the Central Asian community.

Recognising the challenges posed by visa restrictions, the RIPE NCC has taken steps to ensure accessibility for all community members. A notable example is the difficulty faced by our Iranian community members in obtaining visas for RIPE and MENOG meetings. Consequently, with the launch of CAPIF meetings, we have actively encouraged their participation by providing simultaneous interpretation in Farsi and facilitating their integration into the central Asian community, where they can obtain visa-upon-arrival access.

By closely listening to and understanding the needs of different communities, we are able to design regional meetings that cater to their specific requirements. Addressing visa restrictions, language barriers and providing opportunities for easier access to meetings are key factors in fostering a strong and inclusive community. The RIPE NCC's commitment to organising regional meetings ensures that a wider audience can actively participate, contribute, and benefit from the knowledge exchange and networking opportunities these events offer.

The RIPE NCC has a number of initiatives aimed at encouraging newcomers to participate (and keep participating), from supporting academics, to being more diverse and inclusive. We have been collecting metrics to help us quantify how we are performing to understand how we can do better.

Last year we had OKRs on trying to attract 25% newcomers across all RIPE NCC organised events, which include the RIPE meetings as well as the RIPE NCC regional meetings. When we saw that the OKR was met across all our meetings - roughly 32% newcomers at RIPE meetings and over 50% at RIPE NCC Regional meetings - we decided to adjust our OKRs to monitor retention rather than just initiation. At RIPE 86, in the Netherlands, the percentage of attendees attending their first RIPE meeting was 29%, as well as those who came back for their second 13% and third meeting 7%.

We also wanted to gauge the percentage of female speakers at our events. At the SEE 11 meeting in Croatia, we had 27% female participants at the meeting and 30% of speakers on the agenda were female (note that numbers include RIPE NCC staff). At RIPE 86 we had 18% female onsite attendees, and around 14% female speakers across the various plenaries and working groups.

The RIPE NCC also offers a number of fellowships, free tickets to academics and RACI speakers all in an effort to support the RIPE community’s efforts to be inclusive and diverse and see participation from all relevant groups.

Feedback and satisfaction

Turning to how we go about getting feedback on how people in the community use and rate our services, there are a number of approaches we adopt.

One method we’ve been making more use of is to look at our Net Promoter Scores (NPS). NPS provides a useful metric indicating how likely a person would be to recommend a particular service (on a scale from -100 to +100). Scores below zero indicate a higher proportion of detractors, reflecting potential risks and areas for improvement. Scores above zero signify a larger percentage of promoters, indicating strong user advocacy. The wider the gap between promoters and detractors, the more positive the NPS. Scores higher than 0 are typically considered to be good and scores above 50 are considered to be excellent.

We’ve put this approach to use in different ways. For example, to assess whether attendees would recommend our events to their peers, or whether participants in our training courses were satisfied with the services we offer there.

Looking at the NPS scores we collect after each engagement the in person training workshops the RIPE NCC organise receive the highest scores each quarter around 86. If compared to the NPS scores from our online training webinars we see the score drop to around 64.

The dedication and access to the trainers during a three-day in-person training workshop clearly creates a more rewarding experience for workshop participants that outweighs the convenience of attending an online webinar. Something we factor in when we start looking into being cost effective with our engagement with our membership throughout or service region.

Looking at the RIPE 86 meeting NPS, the meeting saw a total score of 49: Promoters 57.1 (36); Passives 34.9 (22); Detractors 7.8 (5).

Another metric we use to quantify the quality of the engagement with the registry is Customer Effort Score (CES), which is a single-item metric that measures how much effort a customer has to exert to get an issue resolved, a request fulfilled, or a question answered.

Our membership engagement team has been working closely with the RIPE NCC’s registry teams to collect the data and report on them regularly. On average the registry received from our membership is around 70+ NPS and 6.5 out of 7 CES.

The RIPE NCC also uses other approaches to better understand the needs and cater from them in the services we offer to our membership and community. My colleague Evelien Schilder has recently written an article about how L&D approaches the needs assessment and quality reviews to their courses.

And the RIPE NCC survey 2023 only just closed last week, with roughly 4,000 responses to process and feedback to analyse.

Quality of our Data Services

As a network coordination centre, we provide tools and data that give insights on the state of the Internet. These services - e.g. RIPE Atlas, RIPE RIS, RIPEstat - provide real value to the community and over the years we’ve received a high number of requests to add to or improve their functionality, to increase coverage, or even to develop new tools on top of those we offer.

While our activities in community and engagement aren’t directly linked to the work that goes into maintaining these services, we are involved in ensuring that the feedback is delivered to the right teams to maintain our reputation as a trusted source of insight and information for the entire community.

You can check out the probe coverage visualisation for yourself on Observable

When it comes to improving RIPE Atlas coverage, the focus should be on quality rather than quantity when distributing probes. Simply sending probes to new networks may seem like a viable solution, but it does not lead to meaningful outcomes in terms of providing users with a comprehensive understanding of the Internet.

The primary goal is to offer RIPE Atlas users the most valuable insights into the Internet landscape. To achieve this, it is important to carefully select networks that serve as advantageous vantage points for probe placement. These networks could include those that cover a significant percentage of the population in specific countries or those located near Internet Exchange Points (IXPs) where crucial traffic flows. By clearly identifying and prioritising these networks, and measuring the progress made in closing the coverage gaps within them on a quarterly basis, we ensure that the request for improved coverage is addressed in a way that truly matters.

To accomplish this the research team has been integrated within the larger community engagement team to establish closer connections with the external stakeholders who will directly benefit from the insights they produce. This collaborative approach ensures that the valuable knowledge generated by the research team is effectively disseminated and utilised by those who can leverage it to make informed decisions.

Drawing from our successful experience with the production of RIPE NCC country reports over the past few years, we are keen to extend our efforts even further. The RIPE NCC is working on developing relationships that will allow us to provide policy makers with accurate and factual insights into the operations of the Internet. By offering policymakers easier access to these insights, we aim to support their decision-making process and facilitate the development of well-informed public policies.

Running out of virtual ink

There is so much more the RIPE NCC can share on our approach to elevating the quality across our various services we provide our membership and the community. Having established a framework for these regular updates makes it easier for us to share more and collect feedback on what we can do to better serve our community needs. I look forward to further engagements on this.