Measuring and Maintaining the Quality of our E-learning

For almost ten years now, the RIPE NCC has been offering self-paced e-learning courses through the RIPE NCC Academy covering topics such as IPv6, BGP Security, Internet Governance and the RIPE Database. The Academy has served over 11,000 new learners since the onset of the pandemic in 2020 and has helped people gain relevant knowledge and skills to contribute to further developing a stable and secure Internet infrastructure.

Since our courses are available from everywhere to everyone 24/7, they serve as great equalisers of knowledge and skills. It’s important to keep in mind that not everyone can access training easily - people have financial constraints, they might be unable to make the time to attend in-person courses, not to mention geographic constraints.

By making our learning material available for free, we are trying to help make the Internet more secure for everyone. Misconfigurations and incidents in someone else’s network caused by a lack of knowledge and skills can affect everybody, not to mention cost time and money to fix.

Focus on quality

We started off with a couple of courses in the Academy, and over the years, we’ve been building our portfolio. We now have over 100 e-learning modules. As this portfolio develops, it’s important to ensure that the quality of these modules remains consistently high.

For years already, learning designers and subject matter experts work together to make sure that our content is not only accurate, relevant and up to date, but that people will actually learn what they’re supposed to by providing relevant examples, scenarios, videos, practice questions and feedback through the learning experiences. Recently we have also started conducting needs assessment interviews (at MENOG 22, SEE 11 and RIPE 86) to learn from RIPE NCC members and the community how we can better serve them and meet their needs as we would like to help solve real challenges people encounter in their jobs.

As the quality of our materials is so important to us, we therefore also wanted to collect better data about the perceived quality of our e-learning courses.

Data about perceived quality

For some time now, we have been gathering statistics on the number of people who enrol in our courses and complete our modules. These completion numbers helped guide us which topics are of most interest to our learners. However, we also wanted to get a more detailed picture of the perceived quality of our content and how each module performs individually beyond completion rates and wanted to make it easier for learners to provide us with such feedback.

Positive feedback in module ratings

Since the beginning of this year, learners can rate each module after completing it. We have already gathered 11,991 ratings from learners with mostly very positive feedback, with modules receiving an average rating of 4.7 out of 5 stars.

We can also see exactly how each individual module performs. So far, each module has received an average rating of at least 4 stars or higher.

Positive feedback related to alignment content with course goals.

At the end of each course, we ask learners to what extent they feel they have mastered each of the course objectives. Similar to the module ratings, learners have generally indicated that the e-learning courses helped them meet the objectives. For most learning goals, more than 95% of the learners (strongly) agreed that the course helped them meet their learning objective.

Gathering open-ended input to improve quality

In addition to gathering this quantitative data, which helps get an overall picture of the perceived quality of each e-learning module, learners can also provide us with open-ended input when giving a module an average or bad rating. The benefit of this open-ended feedback is that it gives us more in-depth and detailed information about their given rating that we can use to further improve the course quality. This has given us lots of practical suggestions that we have already started implementing in our e-learning modules.

Improving our overall L&D processes to handle learner feedback

As we started receiving hundreds of concrete comments per quarter about our e-learning modules, we first had to rethink how we are processing feedback within L&D. We needed to develop a good process and make use of the right tools to triage all incoming comments based on urgency and prevalence, and keep track of all comments over time so they would not get forgotten over time.

During this process, our learning designers are working together with our subject matter experts to synthesise the comments and find concrete ways to improve the courses.

What have we already improved?

Although most learner feedback has been positive so far, learners did have some clear and good suggestions for improvement. One thing we have improved based on their feedback is to make our BGP Security and IPv6 Security lab activities available for people who have the newest Apple laptops with M1 or M2 chips, as this was not possible before. Based on suggestions, we have also made it possible for learners to practise generating BGP filters automatically using bgpq4 in our BGP Security labs.

We have also recently updated four IPv6 Fundamentals e-learning modules by adding clearer question feedback and explanations for parts that were not always clear to some learners.

What are other quality related improvements you can expect?

Other things that are in the pipeline are improvements to IPv6 security videos and the addition of some sample exam questions for courses that prepare learners for a RIPE NCC Certified Professionals exam. This is aimed to better prepare learners to take the exams and smoothen the transition from learning to getting certified.

If you have not already taken a look at our e-learning courses, please check them out! You can take them here and don’t forget to let us know what you think!