Domain ASN Mapper: Understanding Domain Infrastructure

In today's digital landscape, domain reputation assessment has become increasingly critical for cybersecurity, brand protection, and risk management. Organisations need reliable mechanisms to evaluate the trustworthiness of domains they interact with. The Domain Reputation Measure project addresses this need by developing a comprehensive system that analyses multiple dimensions of domain infrastructure to produce actionable reputation scores.

In this direction, understanding the infrastructure underpinning the services and websites is a crucial component, which often requires disparate tools and manual correlation. To address this, we are introducing the Domain ASN Mapper, the first phase output of our broader Domain Reputation Measure project. This open-source tool provides a robust and automated way to map domain names to their corresponding IP addresses and originating ASNs using DNS resolution and BGP routing data. It serves as a foundational component for deeper analysis of domain infrastructure, stability, and ultimately, trustworthiness.

This article details the Domain ASN Mapper's functionality, technical implementation, relevance to the RIPE community, and outlines the future roadmap towards a domain reputation system.

The challenge: Visibility into domain infrastructure

Assessing the trustworthiness or stability of an online entity often starts with checking its domain name. However, the name itself tells little about the underlying network infrastructure. The key questions that arise in providing this assessment include:

• Which networks (ASNs) host the domain's web/mail/name servers?
• Is the infrastructure geographically or topologically diverse?
• Has the underlying infrastructure changed recently or frequently?
• Are the associated networks known for hosting legitimate or malicious activity?

Answering these requires correlating DNS information with BGP routing data – a process the Domain ASN Mapper automates at scale.

Introducing the Domain ASN Mapper

The Domain ASN Mapper is a Python-based tool that seamlessly connects domain names to the Autonomous System Numbers (ASNs) responsible for routing traffic to and from them. It is designed to simplify a multi-stage process that traditionally involves DNS lookups, BGP data analysis, and manual mapping.

Key capabilities: What the Domain ASN Mapper can do

At its core, the Domain ASN Mapper automates a structured pipeline that resolves domains and maps them to their respective ASNs using publicly available BGP data. Specifically, it is engineered to:

• Accept a list of domain names as input, typically in a plain text file format
• Perform comprehensive DNS resolution for each domain. This includes:
  • A (IPv4) records
  • AAAA (IPv6) records
  • NS (Name Server) records
  • MX (Mail Exchanger) records
    (For NS and MX records, it also performs secondary resolution to obtain the IP addresses of referenced hostnames.)
• Map the resolved IP addresses to ASNs and BGP prefixes using a BGP MRT data file, which can be sourced from platforms like RIPE RIS or RouteViews.
• Generate structured output in multiple formats including JSON for APIs and automated processing, CSV for spreadsheet or database import, and Human-readable text for reporting or inspection
• Offer both a command-line interface (CLI) for automation and batch processing, as well as a web-based interface for interactive use.

These features make the Domain ASN Mapper a comprehensive tool for infrastructure analysis, security research, and operational monitoring.

Architecture overview

The system has been developed with a modular design ensuring each core function is encapsulated and independently maintainable. This also supports ease of integration and future feature expansions. The architecture is composed of four primary components:

1. DNS Processor
   Responsible for resolving various DNS record types (A, AAAA, NS, MX) for specified domains.
2. ASN Processor
   Maps IP addresses to ASNs using MRT files.
3. Output Formatter
   Formats the results in multiple formats (JSON, CSV, text) for flexibility in downstream analysis.
4. Web Interface
   Provides a user-friendly browser-based interface for uploading domain lists and MRT files, and visualising the results.

Under the hood: Technical implementation

The Domain ASN Mapper is engineered as a modular Python application, using well-established Python libraries and practices, and is designed for flexibility and robustness.

1. DNS Processor (dns_processor.py):
   1. Leverages the dnspython library for performing iterative DNS queries.
   2. Resolves multiple record types critical for infrastructure mapping: A (IPv4), AAAA (IPv6), NS (Name Servers), and MX (Mail Exchangers).
   3. Handles intermediate lookups: For NS and MX records, it subsequently resolves the A and AAAA records of the returned hostnames.
   4. Implements configurable timeouts and retries (dns.resolver.timeout, dns.resolver.lifetime) via dnspython settings to handle transient DNS issues.
   5. Supports filtering to resolve only IPv4, only IPv6, or both.
      
2. ASN Processor (asn_processor.py):
   1. Utilises the pyasn library for efficient IP-to-ASN lookups.
   2. Requires a BGP data file in MRT format (--mrt-file).
   3. Pre-processes the MRT file into a compressed .dat lookup database (pyasn.mrtx.dump_prefixes_to_file) for significantly faster subsequent lookups. If a .dat file exists, it's loaded directly.
   4. Performs lookups using pyasn.lookup(ip_address), returning the originating ASN and the specific BGP prefix announced that covers the given IP.
   5. Aggregates ASN information across all resolved IPs (A, AAAA, NS IPs, MX IPs), providing a consolidated unique_asns summary including source record type and hostnames for each unique ASN.
   6. Includes error handling for invalid IP addresses and lookup failures.
      
3. Output Formatter (output_formatter.py):
   1. Provides flexible output generation:
      1. JSON: Detailed, structured output for programmatic use.
      2. CSV: Flattened representation for spreadsheets.
      3. Text: Human-readable summary format.
   2. Incorporates error messages for domains where processing encountered issues.
      
4. Interfaces (main.py, app.py):
   1. Command-Line Interface (CLI): Driven by main.py using argparse. Ideal for scripting and batch processing.
   2. Web Interface: Implemented using Flask (app.py). Provides a GUI for file uploads (.txt domains, .mrt/.gz/.bz2 MRT), option selection, processing, and results viewing/downloading. Includes an API endpoint (/api/results/<job_id>) for JSON results.

Workflow overview: From domains to ASN mappings

The Domain ASN Mapper processes data through the following workflow:

1. Input collection:
   1. Domain list (text file) and
   2. an MRT file
      
2. DNS resolution:
   1. Resolves A, AAAA, NS, and MX records
   2. Resolves IPs for NS and MX hostnames
      
3. ASN lookup:
   1. Unique IPs are passed to the ASN Processor
   2. ASN Processor uses the pyasn database (from MRT) to find the ASN and prefix for each IP
      
4. Aggregation and formatting:
   1. Results are compiled, to combine all data per domain including DNS records, IPs, ASN/prefix info
   2. Builds a unique ASN summary with related metadata
      
5. Output:
   1. The Output Formatter generates the results in the chosen format

Scalability and resilience

The tool is optimised for efficiency and robustness: pyasn's pre-compiled database ensures efficient ASN lookups, and DNS resolution includes basic timeout/retry handling via dnspython. The system has been designed so that errors processing one domain do not halt the entire batch, and results indicate errors per domain, while logging provides extensive operational visibility.

In addition, the modular design also facilitate future enhancements such as parallel processing.

Why this matters for the RIPE community

This tool directly leverages and provides insights relevant to the network operators and researchers within the RIPE community. This, in particular, includes:

• Utilising core Internet data
  Relies on DNS and BGP routing data (MRT files from sources like RIPE RIS), making it inherently relevant.
• ASN visibility
  Provides a clear mapping from user-facing domain names to the underlying ASNs responsible for hosting their services. This helps operators understand traffic sources/destinations associated with specific domains.
• Infrastructure analysis
  Enables analysis of infrastructure choices, such as the diversity of ASNs used for critical services (web, mail, DNS) of a domain.
• Foundation for security research
  By mapping infrastructure, it provides a starting point for correlating domains with potentially malicious or compromised ASNs or prefixes.
• Potential integration point
  Future versions could integrate RPKI status or leverage RIPE Atlas measurements for enhanced infrastructure validation (see Roadmap).

Using the tool

The Domain ASN Mapper is available on GitHub.

Installation

Command-line usage

Web interface usage

Refer to the README.md file in the repository for detailed options and usage instructions.

First step towards a comprehensive Domain Reputation System

The Domain ASN Mapper provides the foundational infrastructure mapping. Building upon this, the Domain Reputation Measure project roadmap focuses on integrating diverse datasets and developing sophisticated scoring mechanisms.

Phase 2: Reputation scoring engine and data enrichment

• Goal: Develop the core engine to calculate initial reputation scores
• Data integration: Incorporate BGP/ASN stability history, RPKI validation status, Passive DNS history, WHOIS/registration data, TLS certificate analysis, external threat feeds, and popularity metrics (e.g., Tranco list rank)
• Scoring algorithm: Implement a multi-factor, weighted scoring algorithm, aiming for tunability and transparency
• API development: Create a RESTful API for real-time reputation queries

Phase 3: Advanced analytics, predictive capabilities and community integration

• Goal: Enhance with ML, longitudinal analysis, visualisation, and operational integration
• Advanced Analytics: Implement longitudinal analysis, anomaly detection for infrastructure changes, and explore predictive ML models
• Visualisation dashboard: Develop an interactive dashboard for monitoring reputation scores, trends, and underlying data
• Community and tooling Integration: Explore leveraging RIPE Atlas, establish feedback mechanisms, provide SIEM/SOAR integration points (e.g., STIX/TAXII).
• Methodology and openness: Commit to publishing detailed methodology and potentially open-sourcing further components, presenting findings at forums like RIPE Meetings.

(See the project's README for the full detailed roadmap)

Conclusion and call for collaboration

The Domain ASN Mapper has been developed as the initial phase of the Domain Reputation Measure Project, and offers immediate utility for anyone needing to understand the network infrastructure behind domain names. It automates the crucial first step of correlating DNS information with BGP routing data provided by the community (e.g., via RIPE RIS).

As the foundation for the ongoing Domain Reputation Measure project, this tool, and its future enhancements, will provide significant value for network security, risk assessment, and operational visibility.

We invite the RIPE community and others to test the Domain ASN Mapper, provide feedback, report issues, and contribute to its development. The project is open-source, and collaboration is highly encouraged.

Find the code and contribute on GitHub: https://github.com/soshianse/Domain-Reputation-Measure

Let's work together to build better tools for understanding and securing the internet's infrastructure.