ERRANT: Realistic Emulation of Radio Access Networks

Mobile networks have become ubiquitous, offering connectivity performance from slow connections up to speeds comparable with broadband connections. Having tools to emulate such variability correctly is paramount to test new protocols and applications before publicly releasing them.

Mobile networks have become ubiquitous, but running experiments on them is expensive and challenging, given their complexity and diversity. Emulation can be the solution, and, with ERRANT, we offer a realistic emulator of mobile networks based on a large measurement campaign on European Mobile Network Operators. It improves the current situation, where tools and emulators only implement pre-defined profiles with built-in parameters, which are not supported with real measurements. You can see from Figure 1 how diverse the network profiles of popular applications are.

Figure 1: Table of network profiles for popular applications.

Collecting Data with the MONROE Measurement Platform

Since 2015, the MONROE Measurements Platform has been available to researchers for performing a wide range of experimental campaigns through hundreds of mobile nodes in five European countries. Each node is equipped with three mobile broadband modems, each one connecting to a different operator. MONROE users can schedule experiments to run their experiments on the chosen subset of nodes, as shown in Figure 2. We leverage the results of existing experimental campaigns to build emulation models for the ERRANT framework.

Figure 2: Schematic of measurement infrastructure.

The ERRANT Emulator

The ERRANT emulator is composed of two main components: the emulation model generator and the emulation model exploitation. Figure 3 reports the full pipeline for the first part.

Firstly, we collect the results of Speed Test experiments, each describing the network performance in terms of Latency, Download and Upload bandwidth while downloading content from dedicated servers. Next, we annotate each experiment with the respective Metadata regarding the Radio Access Technology (RAT) (3G or 4G), and the signal strength (Bad, Medium, Good). We aggregate measurements taken under the same network condition in terms of Operator, RAT and Signal Strength, to build the network profiles.

We use data from 100k Speed Tests, creating 32 network profiles: 26 for specific operators and countries and 6 universals. These include data from 2 countries and 4 distinct operators (2 of which are present in both countries). For each network profile, we use the Kernel Density Estimation (KDE) to build emulation models abstracting the 3-dimensional performance metrics that we obtain from the corresponding Speed Tests.

Figure 3: Generation of ERRANT models. Fields with * are used only for specific profiles and not to build universal ones.

For exploiting the models, ERRANT samples the 3-dimension network conditions from the KDE and uses tc-netem to apply them on any network interface. In this way, ERRANT offers realistic network emulation, in which both typical behavior and network variability are accurately recreated.

Validating the Emulation Models

Along with SpeedTest experiments for each network condition, researchers collected a massive amount of independent HTTP measurement campaigns describing the download bandwidth while performing a bulk download of 100MB from dedicated servers. We use these measurements to validate ERRANT models.

In detail, for each network condition, we repeat the HTTP measurement campaigns emulating each network condition using ERRANT, comparing the results with the real ones. Figure 4 shows the effectiveness of ERRANT (the green lines) in the emulation of real mobile networks in terms of average behaviour and obtained variability having a behaviour close to the real measurements (the red lines). Instead, the simple emulation (blue line) with fixed network conditions shows the inability to emulate the network variability.

Figure 4: Comparison of HTTP download speed for 3 models.

Use Cases for ERRANT

Web browsing QoE estimation: ERRANT helps in evaluating the Quality of Experience (QoE) of a user accessing web services. In our experiments, we study the performance of web pages accessed under different network conditions in terms of Page Load Time, also called onLoad time - see Figure 5. Comparing ERRANT (the green lines) with experiment without emulation (the red lines), we notice a strong impact of network conditions, in terms of absolute values and variability. Indeed, a simple static emulation (the blue lines) does not pinpoint the variability of QoE-related metrics that would occur in real mobile networks.

Figure 5: Page Load Time with different types of (emulated) networks.

Adaptive video streaming over HTTP: In recent years, most of the video services (e.g. YouTube) converged towards the use of video streaming over HTTP, in which the server offers the video at different resolutions. We run ERRANT while consuming a video on YouTube and show in Figure 6 how the video resolution at the client has varied. The results show how ERRANT (the green line) allows observing the effect of variable network conditions, causing frequent resolution adaptations. Notice that frequent adaptations are known to be bad for users’ QoE. Conversely, a simple static emulator (the blue line) does not trigger frequent bitrate adaptations, being stuck on 720p after the initial transient period.