Building a Secure Test Environment for an IXP

IXPs operate on principles of trust and cooperation upheld through adherence to clear rules. Quarantine VLAN is a way to help test whether new peers are compliant with those rules. Namex Digital Twin now enables Namex members to carry out these tests for themselves.

IXPs and quarantine VLAN

An Internet Exchange Point (IXP) is a unique and vital hub in the vast ecosystem of the Internet, where network operators from all over the world come together to exchange traffic directly and efficiently. Technically speaking, it works as a Layer 2 broadcast domain — a single subnet where participants' routers connect. But beneath this straightforward definition lies a dynamic and interdependent system that relies heavily on cooperation and mutual trust to operate smoothly.

Imagine a bustling public square where individuals and groups gather to interact freely. Everyone benefits from the shared space, but the environment can only remain functional and safe if everyone follows a common set of rules. In the IXP, this shared space allows members to exchange data seamlessly, but with great freedom comes great responsibility. Without clear guidelines, one participant's actions could disrupt or even jeopardise the operations of the entire community.

To ensure stability and security, every IXP enforces a well-defined set of rules — a code of conduct for its participants. These rules go beyond technical niceties; they are essential safeguards for the entire ecosystem. A key focus of these guidelines is to outline explicitly what types of traffic are not permitted on the peering platform. For instance, harmful activities like sending unwanted packets or malicious traffic are strictly prohibited, as they can compromise the reliability and safety of the environment.

In this way, an IXP becomes more than just a technical infrastructure; it transforms into a community built on trust and cooperation. The rules, far from being restrictive, serve as the foundation for something greater: a shared space where traffic can flow freely and orderly, to the benefit of all.

While we trust that every participant is committed to following the rules, experience has shown that even the most well-intentioned members can make mistakes. A simple misconfiguration can disrupt the harmony of the peering LAN, putting its stability at risk. To mitigate these risks and prevent unwanted traffic on the peering LAN, IXPs implement a variety of protective mechanisms on their switches. These measures include:¹

Blocking spanning tree protocols on peering ports.
Enforcing MAC locking, allowing only one MAC address per connection.
Restricting traffic to specific Ethertypes, such as ARP, IPv4, and IPv6.
Applying storm control to limit the rate of broadcast and multicast packets per second.
Ensuring proxy ARP is disabled on router interfaces.
Discarding link-local protocols, such as IRDP, ICMP redirects, discovery protocols (CDP, EDP, LLDP), VLAN trunking protocols (VTP, DTP), BOOTP/DHCP, etc.

In addition to these safeguards, many IXPs adopt a Quarantine VLAN — a dedicated broadcast domain designed as a testing environment for new members. This approach allows IXP operators to closely inspect a new peer's configuration to ensure compliance with established rules, before granting access to the production LAN.

The process typically involves creating a "dummy" peering and monitoring traffic within the Quarantine VLAN for any anomalies or prohibited behaviour. Some IXPs leverage advanced monitoring systems to streamline this process, making it easier to identify and address potential issues.

But in an era increasingly defined by automation, why stop at manual monitoring? Inspired by Ben Cartwright-Cox’s article², we were captivated by the innovative approach adopted by IX.br, the Brazilian IXP. Ben highlights their “next-generation” Quarantine VLAN testing system — a self-service portal that allows members to run compliance tests on their own.

This groundbreaking solution represents a major leap forward. By empowering members to conduct their own tests, IX.br has not only reduced the workload for NOC operators but also enhanced the efficiency and reliability of the quarantine process. Such an autonomous approach exemplifies how automation can transform traditional IXP operations, improving both scalability and user experience.

Namex Digital Twin and automated quarantine VLAN testing

A few months ago, we embarked on an exciting project: the implementation of Namex Digital Twin. This initiative was a result of our long-standing collaboration with Roma Tre University and its network research group. I have a special connection to this research group, as it was the team I worked with during my master’s thesis, and I knew they had developed a powerful network emulator, Kathará.³ It was clear to us that Kathará could bring our idea to life: creating a virtual replica of the Namex Peering LAN. Thus, the initial idea of the Namex Digital Twin was born.

As the name suggests, Namex Digital Twin is a faithful replica of the Namex Peering LAN, mirroring its members, services (including Route Servers), and even its IP addressing.

What started as a straightforward test environment on the quarantine VLAN soon grew into something much more dynamic and versatile. It allowed our members to test their configurations and BGP setup safely, and it provided us with a way to refine our Route Servers' configurations without impacting the live network.

The idea was that connecting to the Namex Digital Twin was like stepping into the real Namex peering LAN, but with a twist, here, mistakes were not only allowed but encouraged!

It was a safe place where experiments and learning could happen without any consequences for the real network.

From inspiration to implementation

The turning point came when we attended the Euro-IX forum in Crete. There, the IX.br project was presented in detail, giving us a chance to delve into their automated approach. Inspired by their success, we decided it was time to take our work to the next level. We came back home with the idea of integrating automated quarantine LAN testing into our Digital Twin. This new approach aimed to simplify the testing process for members, allowing them to validate their configurations before entering the main peering fabric, ultimately reducing the workload on our team.

Now, the Namex Digital Twin comes with a user-friendly web interface, where members can enter their IP addresses, MAC address, and ASN, and run tests by themselves. This tool automatically checks if a member’s configuration aligns with Namex’s technical requirements. Once a member successfully passes all tests, it can join the peering LAN.

How the Namex Digital Twin works

The heart of the Namex Digital Twin is Kathará, the network emulation platform from Roma Tre University. As mentioned before the first aim of this project was to create a copy of the Namex peering LAN, its "digital twin" indeed. So our approach does not just emulate Route Servers; it creates a Docker container for each member that is part of the production environment.

This means that the emulation is not a simplified model, it mirrors the real world down to the details. Route Servers run the same BGP daemons (Bird and OpenBGPd) and use the same configurations as those on the main peering LAN. Essentially, we have created a real-world scenario but within a separate collision domain: the Quarantine VLAN.

Here’s how it all works. The network scenario starts by taking a snapshot of the Namex Peering LAN, but the environment is not static. We provide scripts that keep the scenario synchronised with the live network, using real-time data like:

Dumps of the actual Route Servers’ RIBs (IPv4/IPv6)
The real Route Servers’ configuration files
The export of all the members connected to the main peering LAN (we use the IXPM members export)

A Python script manages the reloading of Route Server configurations and handles the creation or removal of member containers accordingly.

A closer look at the Quarantine Dashboard and its validation checks

We've recently enhanced our system by adding automated quarantine checks and a user-friendly web interface. Through this interface, members can run tests on their configurations and review the results directly. Configurations can be tested and refined before they touch the live network. The purpose of the dashboard is simple: to ensure that a peer’s configuration is thoroughly checked before they’re allowed onto the production LAN. We encourage peers to bring their actual configurations into this testing environment, using the same setup they intend to apply in the live environment, so that any potential issues can be caught early, before impacting the real network.

When a member initiates the tool through the web interface, it starts running a series of automated checks.

Now, let’s dig into the details and take a closer look at the checks that we’ve introduced so far, and how each one helps ensure a smooth transition to the production LAN:

Connectivity tests:
- Ping/ping6: Tests basic connectivity using ICMP ping, ensuring that the peer can reach other devices on the network using IPv4/IPv6
- ProxyARP: Verifies that Proxy ARP is disabled on the interface to avoid unwanted routing of packets, ensuring that each peer handles only the traffic intended for them
- MTU: Checks the MTU settings to make sure that packet sizes are correctly handled

BGP checks include:
- Established sessions: verify the BGP session between the peer and the Route Server is properly established
- Prefix limit check: Ensure that the number of prefixes advertised by the peer does not exceed the allowed limit
- Default Route Advertisement: Check if the peer is announcing a default route (0.0.0.0/0 for IPv4 or ::/0 for IPv6)
- Private Prefix Advertisement: Verify that the peer is not announcing private prefixes that should not be routed through the IXP.
- Next Hop Validation: Ensure that the next hop of advertised prefixes corresponds to the peer's router IP, as assigned by the IXP
- AS Path Consistency: Confirm that the AS-PATH of the prefixes they advertise starts with the ASN of the peer
- Announcement Consistency: Check for differences between the announcements made to Route Server 1 (RS1) and Route Server 2 (RS2)

Unauthorised traffic detection: Monitor traffic for 1 minute to detect any unauthorised traffic types such as neighbour discovery protocols, internal routing protocols, or unsolicited router advertisements.

Security tests: are conducted to detect open ports for DNS, NTP, SNMP.

If a member successfully passes all the checks, it means he is ready to join the Namex Peering LAN and he can contact the NAMEX NOC. What happens if some checks fail? The member needs to address the issues, adjust the configuration, and then run the test again.

Conclusions

The Namex Digital Twin will be now available to our customers, and we believe it will play a key role in enhancing the security of our network. We hope that this tool will encourage our members to test their configurations thoroughly, making the peering environment safer for everyone.

However, it’s important to note that unwanted traffic needs continuous monitoring, as router configurations can change frequently. Keeping an eye on this ensures that the network remains secure and stable. Of course, for large-scale IXPs, replicating a complete copy of their network can be a significant challenge. The complexity and scale of their infrastructure make it difficult to create an exact digital twin, requiring considerable resources too.

But the key takeaway is this: no matter how you choose to test, automated testing can enhance the security of the peering LAN, or at least, it represents a step in the right direction toward a more reliable network.