Challenges in Identifying Remote and Indirect Peers

IXP operators do not always know or publish their members that are remote; even the available data (for example, in PeeringDB) can be unreliable.

The main approach to identify Remote Peering is to measure the RTT between a vantage point inside the IXP and the IXP peering interface of a member. If the RTT is above or below a certain threshold, typically 10ms, then the member is considered as remote (local). However, in today’s IXP landscape latency alone does not suffice:

1. Using a single latency threshold heavily misinfers RP. Our measurement study shows that a threshold of 10ms infers 40% of the remote members as local. Reducing the threshold to 1ms still misinfers 18% of remote members, while reducing it further would misinfer local members.
2. IXPs are not monolithic infrastructures. Instead, many IXP operators distribute among multiple co-location facilities across multiple cities or even countries. A member Autonomous System (AS) may be physically present in one of the IXP’s facilities and still have high latency to reach another IXP location. The Delay Matrix of NETIX, a wide-area IXP distributed across 22 European cities, illustrates that the pairwise latency between its facilities ranges from 1 to 60ms.

Peering Aspects

To address the earlier mentioned challenges, we introduce a ‘first-principles’ approach which combines RTT measurements with technological and economic aspects that define the peering paradigm.

Port capacity: Typically, physical port capacities range between 1GE and 100GE. Connecting through a reseller allows IXP members to purchase virtual ports at fractional capacities at lower prices compared to physical ports. Since port capacities lower than 1GE can be obtained only through resellers, we use it as an indication of indirect peering.

Presence at co-location facilities: To establish a direct connection (local peering) to an IXP, an AS needs to deploy routing equipment in at least one co-location facility where the IXP has deployed switching equipment. We collect the facility list (including geographical coordinates) from PeeringDB and Inflect, and the websites of the 50 IXPs with most AS members. IXP websites provide additional facility data for 48% of the IXPs, allowing us to compile a more comprehensive dataset.

Assuming perfect knowledge of port capacities and co-location data, identifying RP would be a straightforward lookup process. However, the available data are incomplete and noisy. To resolve such cases, we investigate two additional routing features.

Multi-IXP routers: An AS may connect to multiple IXPs through the same border router to reduce operational costs; we call such routers, multi-IXP routers. We distinguish three cases:
(i) When multiple IXPs are present in the same facility, a co-located AS may connect directly to all of them using a single router.

(ii) Remote peers may connect through the same reseller to multiple remote IXPs where this reseller is present.

(iii) An AS may connect with the same router to both local and remote IXPs if, for example, it is co-located with one IXP and uses a reseller for another.

Private connectivity: Two networks co-located at the same IXP-hosting facility can interconnect directly with each other (private peering) without using the IXP infrastructure, for example, to reduce costs in case they exchange large volumes of traffic. When an IXP member appears to be privately connected with several ASes co-located at the same facility, this is a strong indication that this member is local to that facility. 

To identify multi-IXP routers and private peering interconnections we analysed traceroute measurements collected through the RIPE Atlas platform. Multi-IXP router interfaces appear in different traceroute paths to be interconnected with different interfaces of different IXPs. Instead, private peering practices are identified by searching for interconnections between interfaces of different ASes without any intermediate IXP interface. To infer IXP interconnections in traceroute paths we use the traIXroute tool.

Inference Algorithm

Step 1: Find reseller customers via port capacities. For each IXP member, we obtained the capacity of its peering port(s), through the IXP website or PeeringDB. If the member’s port capacity was lower than the minimum capacity of the physical ports reported in the pricing section of the IXP website, we inferred that it was connected to the IXP through a reseller. We were then able to identify it as an indirect peer.

Step 2: RTT measurements. We compiled a list of publicly accessible Looking Glass servers (LGs) and RIPE Atlas probes which reside inside IXPs and whose exact location is known. From these Vantage Points (VPs) we repeatedly pinged the IXP IP interfaces of each IXP member every two hours for two days; to reduce sensitivity to network conditions we store the minimum RTT value for each interface.

Step 3: Co-location-informed RTT interpretation. For each IXP interface, we calculated a geographical area around the VP location where the IP interface of the IXP member can be located based on the minimum RTT. To calculate this area, we estimated the maximum and minimum distance between the VP and the interface based on the RTT. The presence/absence of a facility of the IXP in this area denotes a local (remote) member. 

Figure 1 shows a measurement from a VP in NL-IX in Amsterdam to an IXP member. The green-shaded ring denotes the feasible geographical area where the member can be located, according to a minimum RTT measurement of 5ms (d1 is the largest possible distance and d2 the shortest). By checking the facility data both of the IXP and its member we determined that it is located either in Frankfurt or London. Therefore, even for distributed IXPs we can infer members that are directly connected to the IXP despite high RTT measurements.

Figure 1: Combining RTT measurements with co-location data to infer the possible location of IXP members.

Step 4: Multi-IXP router inference. The previous steps may not be able to infer the peering type due to missing facility data or RTT measurements. In such cases, we used the multi-IXP router feature. We parsed the traceroute paths, and for all the IP interfaces of the same AS that are connected to IXP interfaces we performed alias resolution to group them to routers. If we found a router connected to multiple IXPs, and we had inferred the AS as local or remote to one of these IXPs from a previous step, we extended the inference to the rest of the involved IXPs based on whether they share co-location facilities or not.

Step 5: Localisation of private connectivity. If an AS had private peerings over the same router through which it is connected to an IXP, and the private peers are physically co-located to IXP facilities, we inferred that the AS is also local to the IXP.

90% of IXPs Serve Remote Peers

We executed a ping measurement campaign from 7-9 April 2018 from each LG and RIPE Atlas VP to the peering interfaces of the IXP that hosts the VPs. For the same period, we parsed the publicly available RIPE Atlas traceroutes, and applied our methodology. Figure 2 shows the final inference results for the 30 largest IXPs in terms of the number of member ASes. Overall, we found 28% of the IXP interfaces for which we made an inference to be remote, with 90% of the IXPs having more than 10% of their peers as remote. Moreover, almost 40% of the members of the two largest IXPs (DE-CIX and AMS-IX) are remote.

Figure 2: Inference results per IXP; Figure 3: Minimum RTT for each responsive IXP peering interface.

As shown in Figure 3, 75% of the IXP interfaces are within 2ms from the respective VP, while more than 20% of the interfaces have minimum RTT over 10ms, a 2-fold increase since 2014. Interestingly, 6% of the peers inferred as remote/indirect claimed local presence in at least one IXP facility. Based on RTT results and port capacities, almost 60% of these peers are indeed local in an IXP facility but prefer to connect through a reseller, while for the rest 40% the claimed facility presence is likely spurious.

Methodology Yields ~95% Accuracy

To validate our results, we obtained data from IXP operators through direct communication and IXP websites. Note that IXP operators usually only know whether their members are connected through resellers, and not where they are located, or if they use a L2 carrier to access their co-location facilities (a primary motivation for our study). As shown in Figure 4, our approach yields ∼95% accuracy and precision, and the results are consistent across IXPs.

Figure 4: Validation results per IXP in our test validation dataset; Figure 5: The increase of remote peers is 2x faster compared to the increase of local peers.

Preliminary Insights and Next Steps

To understand the evolution of RP over time, we collected longitudinal data between 4 July 2017 and 10 September 2018 for 5 IXPs (LINX, HKIX, LONAP, THINX, and UAIX) and calculated the aggregate growth and departure rates per peering type. In Figure 5, we observed that the number of remote peers grows twice as fast as the number of local peers. However, remote peers also exhibited higher (+25%) departure rates than local ones. These results indicate that IXPs that service most of their country-level peering ecosystems mainly expand their market pool by attracting remote peers. However, such peers do not commit substantial resources, making it easier for them to terminate their IXP connectivity.

In terms of RP impact on resilience, we found that 20% of the remote peers rely on multi-IXP routers, meaning that the AS- and IXP-level peering diversity of such peers are misleading indicators of their resilience. This is because all of their interconnections depend on the same physical equipment. 

As future work, we plan to:

(i) investigate such resilience implications in depth,

(ii) examine traffic ratios of remote versus local peering interconnections at IXPs, and

(iii) scale up our inference methodology by extracting RTTs from traceroutes (crossing IXPs) instead of pings (from within the IXPs).

For more information on the work please read the full paper published at IMC 2018 and watch our presentation at RIPE 77. You can also leave us a comment below.

A demo of our methodology and relevant data are available at http://remote-ixp-peering.net

If you like to help this work, please contact us and consider contributing validation data and/or feedback.

Acknowledgements: This work has been funded by the European Research Council grant agreement no. 338402.