Assessing the Risk of New .nl Registrations Using RegCheck

Around 15% of the abuse reports we receive for .nl involve a domain name registered less than 30 days prior to the report. Since the domain names are reported so soon after registration, one can assume they were registered with malicious intent. Therefore, by automatically identifying and stopping malicious domain name registrations, we can improve the safety of .nl. In this blog, we introduce RegCheck: a system that assigns interpretable risk scores to new domain name registrations. We describe RegCheck’s requirements and design, show it performs with 48% recall and 22% precision when applied to historical data, and explain how our abuse analysts have been processing potentially malicious domain names identified by RegCheck.

Fighting maliciously registered domain names

Malicious domain name registrations are a blot on the landscape: even though only 0.15% of the newly registered .nl domain names are reported within 30 days of registration, this small group accounts for as many as 15% of all .nl-related abuse reports. For this reason, some fellow registries are already identifying potentially malicious registrations. Similarly, the EU’s NIS2 directive is contributing to the fight against malicious use of domain names by introducing – amongst other things – new regulations to enforce the verification of domain name registrant data.

At SIDN Labs, one of our graduate students published a thesis on identifying potentially malicious domain name registrations in early 2021. We continued his research and wrote a feasibility study in March 2022. The results of that study were promising, so we started a search for a mature system to automatically identify high-risk registrations, which we define as domain name registrations having a high likelihood of hosting a malicious website in the future; for example, involving a phishing scam.

Our primary goal is to design and prototype a system that we can use to identify high-risk .nl domain names at the time of registration. Therefore, identification can only rely on the data available at the time of the attempted registration, such as the domain name itself and the registrant’s postal address. Information about the eventual website or other applications hosted on the domain name will not become available until later and is therefore out of scope.

The secondary goal is to implement a system that fellow registries can also use to assess the risks of new registrations they observe. Since different registries have different abuse policies, in this blog we focus on identifying high-risk registrations, and ignore possible responses, such as deferring the domain’s entry in a registry’s DNS zone.

This blog introduces 5 requirements and analyses existing solutions to test those requirements. We conclude that none meets our requirements. We therefore introduce RegCheck – the system we have developed in conjunction with our abuse analysts – and subsequently scientifically assess whether it satisfies our requirements, by performing quantitative analyses of the system’s accuracy, for example. We conclude this blog with our plans for 2023.

Requirements

Requirement 1: Accurate

An incorrect risk score can have serious consequences. A malicious registration that goes undetected due to a low score can harm internet users, though we do not want to trouble abuse analysts or registrants with many false alarms. Therefore, the system must be as accurate as possible.

Requirement 2: Interpretable

Although responses to potentially malicious domain names are outside the scope of this blog, the generated risk scores will certainly have consequences for domain name registrations and their registrants. For example, registries may defer or even cancel high-risk registrations. Since high scores will affect the registrant and possibly other stakeholders, we believe it is vital that we know the grounds on which the system generates a score. Therefore, the system’s scores must be interpretable and tie in with insights from the field of explainable AI.

Requirement 3: Straightforward

We prefer straightforward methods because complex systems such as deep neural networks usually require more resources and are less interpretable – while not necessarily resulting in better performance. Therefore, the system must use the simplest possible method that works.

Requirement 4: Registry agnostic

Our secondary goal is to create a system that fellow registries can use. Given that different operators employ different policies and registration databases, the system must be registry agnostic. For instance, it must be able to connect to various registration databases.

Requirement 5: Customisable

A registry operator might want to explore different approaches. The system should therefore be customisable; for example, by giving operators the freedom to include or exclude risk factors depending on the type of data that is available to them.

Existing solutions

Some of our fellow registries already use systems for detecting potentially malicious registrations. We discuss 4 systems that provide a good overview of the breadth of approaches currently deployed at European registries.

Firstly, EURid (.eu) has a thoroughly evaluated system called Premadoma, which relies on machine learning. Premadoma focuses on registrations associated with large-scale malicious campaigns, but SIDN also wants to detect scammers attempting to register a single suspect domain name.

Secondly, DNS Belgium (.be) and SWITCH (.ch, .li) currently employ a scoring system that relies on static rules to pick up potentially malicious registrations. These systems are straightforward and interpretable, but their static nature limits the degree to which they can be customised. DNS Belgium is exploring the possibility of using machine learning. We have identified some commonalities and are in close contact with them.

Finally, Nominet (.uk) developed Domain Watch, which targets potential phishing scams and suspends potentially malicious domain names shortly after registration. We know little about Domain Watch’s inner workings and performance, because Nominet has chosen to be cautious with what it places in the public domain, for understandable reasons. We have nevertheless opted to share RegCheck's design for transparency, and to enable other registries to also use our approach. However, as Nominet does with Domain Watch, we keep our risk factors private to prevent adversaries from working around them.

RegCheck design

Since none of the existing solutions met all our requirements, we decided to develop one ourselves. We call this system RegCheck (short for registration checker) and show its basic design in Figure 1. The system contains a core, registration connector, an internal database and 4 command-line interface (CLI) programs that allow operators to interact with the system.

The core allows registries to create and evaluate statistic models that assign risk scores to domain name registrations. A risk score is expressed as a percentage indicating the likelihood that an abusive website would be hosted using the domain name.

We currently provide 2 base models for estimating risk scores: knowledge-driven and data-driven. Both are linear models that calculate risk scores by looking at risk factors individually. Currently, we consider eleven risk factors, such as suspicious character combinations in the domain name and inconsistencies in the registrant data. However, the knowledge-driven models require a human expert to determine the impact of each risk factor on the final score, while the data-driven base model employs logistic regression – a machine learning algorithm – to assign weights to the risk factors using labelled registrations. Labels can be added automatically from abuse feeds, or manually from a CSV file. Furthermore, it is possible to customise the base models by implementing additional risk factors or by creating a completely new base model; for example, by using a different machine learning algorithm.

Registrations are fed into the RegCheck core through a registration connector, or manually from a CSV file. A connector links RegCheck to the registry’s registration database and is used to import subsets of domain name registrations. The registration connector must be implemented by each registry themselves, because of the large differences between registration database implementations. The only requirement is that the connector publishes an API the core can connect to.

RegCheck Evaluation with SIDN’s abuse analysts

We have been applying RegCheck to newly registered .nl domain names since August 2022 and presenting domain name registrations with a score above a predefined threshold on a dashboard. Our abuse analysts review these registrations regularly and judge whether they indeed pose a risk. When they do pose a risk, our analysts ask the registrant to prove their identity, and the domain name is taken offline if the registrant does not comply with that request within the time set in Articles 16 and 18 of the general terms and conditions for .nl registrants.

Our abuse analysts are positive about RegCheck and have provided us with useful feedback. However, since interventions fall outside the scope of this blog, we will not evaluate the interventions taken by our abuse analysts. Instead, the section below assesses whether the system satisfies our requirements.

Qualitative evaluation of Requirements 2-5

As mentioned earlier, we require a system that generates interpretable scores (Requirement 2). Linear models derive their results by adding up the effects of all individual features. That makes the model interpretable, as we can assess the contribution of each individual risk factor to the final score. The linear nature of our models allows us to calculate the relationships between scores and risk factors associated with a domain name registration. That makes scores intrinsically interpretable. Figure 2 shows an example of how we explain a score to our abuse analysts by indicating the risk factors that contributed to the score.

Requirement 3 is that the system must be straightforward. Our models are straightforward, because they consider a small number of risk factors and only consider linear relationships. Furthermore, RegCheck provides a CLI program to automatically import labels, train models and apply them to new registrations. Deploying RegCheck therefore requires little effort, which means that Requirement 3 is met.

Requirements 4 and 5 relate to our secondary goal of creating a system that fellow registries can also deploy. We can only guess to what extent our design meets these requirements, because we have not yet validated our system with another registry. Firstly, we believe RegCheck is registry agnostic, because it can import data from various registration databases (Requirement 4). Secondly, risk factors can be omitted if desired, and new factors and base models can be implemented. That makes RegCheck customisable (Requirement 5).

Qualitative evaluation of Requirement 1

Next, we analyse RegCheck’s accuracy. We start calculating the system’s accuracy against historical .nl data and later monitor its accuracy ‘in the wild’ when being applied to newly registered .nl domain names.

Accuracy on historical data

Methodology

For this analysis, we create 2 risk assessment models. The first is a knowledge-driven model and uses static rules that we designed in conjunction with our abuse analysts. The second model is data-driven and employs logistic regression. We fit this model using 2100 malicious registrations and 103k random legitimate registrations from February 2021 to August 2022. Malicious in this case means .nl domain names reported on Netcraft’s abuse feed within 30 days of registration. Registrations not reported within 30 days are considered to be legitimate.

We also craft an evaluation dataset containing 341 malicious registrations and 8700 random legitimate domain names from August to November 2022.

The last preparatory step is choosing the threshold value, which determines the score above which a registration is classified as being high-risk. In consultation with our abuse analysts, we have set the threshold values based on the number of registrants’ identities our analysts can currently manually verify in addition to their day-to-day work, which is about 10 registrations daily.

Results

We summarise our results in Table 1. We see that the knowledge-driven model has a recall of 9.38%, while the data-driven model detects 47.80% of the high-risk registrations in our evaluation dataset.

Furthermore, we include the positive predictive values (PPVs), which provide a good proxy for the models’ precision when deployed in production. The knowledge-driven model and data-driven model obtain PPVs of 0.55% and 22.08% respectively, keeping in mind a prevalence of 0.15% malicious registrations, which is the percentage of all .nl domain names reported within 30 days. That means 0.55% of the registrations identified by the knowledge-driven model are indeed high-risk, while 22.08% of the data-driven detections involve a high-risk registration.

Although those results demonstrate that RegCheck is not perfect, they indicate a satisfactory balance between detecting potentially malicious domain names and wasting time on responding to too many false alarms. In addition, we believe significantly higher results are not realistic due to the limited information that is available at the time of registration. Given that the performance of the data-driven model far exceeds that of the knowledge-driven model, we will continue with the former during the rest of our analyses.

Accuracy on new registrations

We first evaluated RegCheck’s accuracy when applied to historical data. That gave us enough confidence to deploy the system in August 2022. In this section, we evaluate RegCheck on all newly registered domain names from 17 November to 8 December, the period during which the most recent version of RegCheck was deployed.

During that period, 43k domain names were registered, of which we placed 181 on the RegCheck dashboard, because their risk score exceeded the threshold. On average, therefore, we published 9 domains daily – about the number we expected.

Our abuse analysts reviewed the published registrations and judged that 38 did indeed pose risks. That yields a precision of 21%, which is in line with the precision obtained when applied to historical data. We do not observe any correlation between the score and judgements. In other words, we would not expect a higher precision to result from increasing the threshold.

Conclusions and future work

Over recent months, we have created RegCheck: a risk assessment system for domain name registries that is accurate, interpretable, straightforward, registry agnostic and customisable. We have been using the system successfully since August 2022 to identify high-risk .nl domain name registrations, which means our primary goal has been achieved.

We would like to highlight that SIDN does not stop at assessing the risk of domain name registration. Our abuse analysts act on high-risk registrations identified by RegCheck every day. As an organisation, we continue to think about how we can further improve our approach to tackling potentially malicious registrations, partly with the NIS2 directive in mind. We will report on further progress in a future publication.

Finally, we will start a joint project with DNS Belgium (.be) this year. As mentioned earlier, they also have experience with identifying high-risk registrations and we therefore want to evaluate each other's systems and see if we can learn from each other. Furthermore, we would like to hear if other registries also want to collaborate on this topic or evaluate our system.

This article was originally published over on the SIDN Labs blog.