Interconnection Agreements at Scale: Secret or Simple?

By Martin Hannigan and Job Snijders

Settlement Free Interconnection “SFI” (FCC, 49, 98)  is a way of life for some Internet networkers. It is how the business of the Internet works. It is also a craft. Some view it as a dark art (Lohdi, 1), where it is impossible to learn what happens behind closed doors. Who is in that smoky backroom (Lohdi, 1)? Network operators of all kinds. What is being discussed? Business. Is it difficult to learn? It can be. While it is true that the world of SFI and the agreements that make it happen are secretive (Brodkin, 1), it is also possible to peek behind the curtain. If you know how to look. Of course, a few pointers can be useful.

SFI is both a business and technical relationship between two networks. It is an agreement where the networks concede to trade traffic between each other’s customers. Without payment or without “settlement”. There are multiple steps to creating an SFI relationship. There are also many ways to walk down those steps. To keep things simple, we are going to look at only Private Network Interconnection “PNI”. Other forms of SFI (e.g. peering over a public Internet Exchange) are valid, but the majority of traffic on the Internet flows over PNIs. Without private connection based SFI, the Internet would be centralized across intermediate platforms. Risk would be higher. Performance could be lower. Costs would be substantial. By using the SFI approach, we fire on all cylinders. We reduce risk and cost widely.

SFI relationships require more than just knowledge of how BGP works. Business realities must be understood and mixed skillfully into the recipe. This includes technical aspects to ensure the proper outcome. Simply throwing fiber over a cage wall worked 25 years ago. Today, lawyers, accountants, project managers and performance testers are stakeholders. They are part of the conversation. They add collective value to the business decision. As a collective, they ensure that an agreement is as comprehensive and practical as possible.

Not all SFI agreements are created equal. Many are based on handshakes or email. Some are small memorandums that document non-binding terms. Others are intense, containing much legalese. The agreements can be complicated. Some are more prescriptive than you would expect. Every business has standards and they differ. It is not shocking or odd that the agreements are different. The more complex agreements seek to set a cadence of predictability.  The agreements support business and network security, stability and resiliency. The most useful agreements are fair and executable, based on the ideology of mutual benefit.

Objectives of an Interconnection Agreement

An SFI agreement defines the scope of interconnection activity. While it is impossible to think of every benefit, a broad tone can be set. Intent is important. It sends a message that the parties will act mutually. Even in the event of silence on a matter that may arise later. Networks using SFI agreements need to spend time thinking about what they want and why, understanding how an agreement affects the future and benefits.

Objectives for an interconnection agreement to consider:

• Provides for cost savings and performance improvements
• Ensure the exchange of traffic is secure, stable, and resilient
• Establish timely cooperation for security and network incidents
• Grant network traffic control rights
• Usually includes a non-disclosure agreement
• Embraces The Law of Least Astonishment

The Law of Least Astonishment (Geoffrey, 1) should be a goal of any agreement. An uneventful SFI arrangement is always best. It is also not always possible. While you may find that you can create a long list of needs, it is efficient to mimic what has succeeded in the past. Smaller can be better. Generalities can be useful to capture scope. Cooperation on security and routing events makes the parties stronger. Enabling capital management and planning is a priority. In reality, our world of Internet networks is anything but predictable. It is important to achieve a level of consistency where possible. The SFI approach does its best to support this.

Business terms are needed as part of the objective. Business terms are negotiated by the network owner. Normally, by the team of engineers and others responsible for the health and welfare of the network. They negotiate utilization, capacity and management parameters. Legal terms are negotiated by lawyers. While plain language (Bradner, 1) is always best, legal language is what makes it an enforceable agreement. Networks should cover all the necessary business, technical and legal points in the scope, such as term, jurisdiction and venue. Include all necessary parties in the conversation. Including a wide audience in the conversation helps to set realistic business goals. Educating people all along the process also makes it easier to get buy in.

General Requirements

Having a list of prepared requirements is the first step to begin a conversation. It sets expectation and intent. It describes the methods and defaults of how you will interconnect. There are no surprises here. The requirements are common and well understood. Start with requirements that are written down. From there they can be tweaked and agreed to in writing.

Requirements included in many SFI agreements:

1. Peers must operate an Internet Protocol (IP) network and use Border Gateway Protocol version 4 (BGP-4 as defined in RFC 4271 and updates) to manage the exchange of traffic at the Interconnection points for all Internet traffic
2. Peers must operate a managed 24x7 NOC and must remedy reported issues within a reasonable time-frame
3. Peers must use mutually agreed Autonomous System Numbers (ASNs) at each Interconnection point and announce a set of routes which should be consistent with publication in an IRR
4. Peers must announce all routes including customer downstreams
5. Peers must apply routing filters facing their customers, to ensure correct announcements from its customers
6. Peers must agree not to abuse the Interconnection.
7. Activities such as pointing a default route at the peer or otherwise forwarding traffic for destinations not explicitly advertised, resetting next-hop, selling or giving next-hop to anyone is forbidden
8. The scope of an Interconnection arrangement can be global or localized to a specific region.
9. Public peering interconnection by Internet Exchange Point “IXP” should be converted to private network interconnect (“PNI”) once the parties reach 15% or higher average utilization for about three months
10. Peers should make upgrades at the 95th percentile utilization rate of 65%, measured over a rolling three month period
11. Parties should provide each other with access to a BGP Looking Glass to enable debugging operations
12. Peers agree to perform capacity reviews at least once per year if not more as needed
13. Each network absorbs their costs
14. Cross-connect fees equitably split on PNIs
15. Maximum latency to the next-hop should be no more than 5 (or X) milliseconds RTT. If the peer shows higher round-trips, the peer might be engaging in remote peering, which raises a red flag: “is this the right place to interconnect?” “why are you transporting this so far?” Exceptions can be made. Transparency matters.
16. Peers should agree on a common automation source for all add/move/drop peering, for example PeeringDB.

None of the requirements above are set in stone.  They are business terms you would seek to negotiate. Some networks have higher or lower defaults. It can take time for a network to do an upgrade. Parties will tweak parameters to address provisioning, capital planning and other needs. Security may be more important and you’ll set an expected response time for a priority issue. You might set the PNI utilization rate to 70%, but lower the measurement that triggers an expansion PNI. There could be language about new traffic and immediate demand. One might argue that for sudden needs to be accommodated, the other party might trade something else of value. The SFI dance is a tango of two networks. While it is not always beautiful, it usually works.

Common Blockers

It is likely that you will encounter blockers in conversations with a few networks. Some networks publicly post the conditions of interconnection agreements (Comcast, 1). Some interconnection agreements state that if one meets the conditions, they still may not interconnect (Comcast, 3). Others do not publish their requirements (Cogent, 1). Many continue to rely on conditions that were useful a decade ago and in a way that reflected the traffic patterns of the day. While these conditions may continue to be relevant, they can sometimes be more onerous as a result of being less than important. All networks are different. While some may have good reason to insist on a condition, others will be more pliable and flexible. Your mileage is going to vary. Whatever approach you take is squarely up to you. If you choose to object to a condition that may be a blocker, research it. Gain confidence. When you make your opinion known, on either side of the table, have confidence in your belief. It should only serve to convince a party that you know what you are talking about.

Examples of common blockers used to deny interconnection requests:

Downstream ASN Demands 

Some interconnection requirements include the demand for a (large) number of downstream ASNs. If there is mutual benefit to the interconnection, the number of downstream ASNs is not relevant technically speaking. A peer could deem it relevant for some other reason. Its relationship to traffic is null.

Backbone Requirements

Some networks use the Internet instead of a physical backbone. While a backbone and the resulting network control has benefits, what matters most are the benefits of the traffic exchanged. If a party can control their own traffic by localizing it and demonstrate mutual benefits, what is the harm?

Lack of Balance (Ratios)

Some peering agreements include requirements that the amount of traffic exchanged in each direction be balanced. For instance, traffic in one direction cannot be more than double the traffic in the other direction.

This requirement was originally justified because hot-potato routing and centralized web servers caused unbalanced traffic flows. In today’s Internet of distributed content, CDNs, edge caching, and other techniques, the requirement for balanced ratios (Steenbergen, 29) may not be applicable.

At the end of the day, SFI is a business tool to further the goals of the corporation(s). If a network can increase revenue or decrease costs by turning up an SFI link, staff would be doing their company a disservice by not agreeing to interconnect. If there are performance, income or expenses benefits to be gained, it must be taken seriously in the interest of stakeholders and shareholders. However, when the business case fails, it is fair to adjust the parameters. Fairness is key, as any large inequities will later show themselves and degrade the benefits over time.

Workarounds to Interconnection Agreements

Refusing to interconnect can be cultural. There may be a fear of the regulator. If after all these years of fighting interconnection, a network suddenly changes their ways it can create concerns. It can peak interest by shining a spotlight on current practices. If other customers hear of another getting an SFI relationship, it could create a “run on the bank”, sparking chaos for multiple relationships. It may also impact their value. External misunderstanding of network relationships has a wide variety of impacts. During periods of consolidation in the industry, interconnection agreements can impact value or affect an acquiring party in unexpected ways. Such as acquiring a settlement free relationship of a customer. While getting around these concerns are trivial, you don’t get the same benefits security as you would with a formal agreement.

Existing service agreements can mimic peering and interconnection agreements. Parties can add, drop or change terms to an MSA on renewal. Who cares what it is called as long as the same benefits are provided? A party can add an amendment or other equally innovative language that results in benefits being provided. It enables the parties to interconnect without a formal interconnection agreement. It documents the interconnection and becomes the agreement. It may not be as detailed as we would like, but it serves as the same assurance. It is a baby step in the right direction if a formal declaration is desired in the future.

Example negotiable language that can be added to a services order:

• Upgrades required at 65%
• Parties must meet regularly to review interconnection
• 24 x 7 x 365 direct NOC access required
• All routes including self and down streams
• Language about on-net vs off-net billing characteristics {ports, traffic, bits, access}
• Must include all reasonably expected routes

When to Use an Interconnection Agreement

Interconnection agreements aren’t for everyone or all situations. They can create unnecessary legal overhead. Handshakes work when you are not overly concerned with the loss of an ASN’s traffic via a direct PNI. Peering at exchange points is usually done by handshake and should remain that way. Larger, private interconnection is different. Networks which have traffic that if lost would dramatically increase costs or substantially reduce performance should look to formalizing the relationship with an agreement. Keeping overhead low is important which is why scale is applicable when determining the need for a formal agreement. When importance outweighs overhead, you know it is time to formalize.

Examples of when to use an agreement:

• Large volumes of traffic significant to a network’s business results
• To protect capital and operational expense and investment in the relationship
• Seeking to assure traffic and growth for business planning
• Ensuring access protections in the event of M&A
• Establishing terms of mutual benefit(s) and what they are
• Performance maintenance
• When a relevant party asks you to

How Hard is this Going to Be?

Asking for private interconnection can make or break your credibility and the network relationship. It is important to warm all parties to the idea. It takes an effort to start and finish an agreement. While it might be tempting to blurt a request at the other side, it is not always helpful. Avoid blindly sending email to a peering@ email alias. The shock and awe factor never helps. It also creates a record. The other party may be amenable to finding another way to achieve a common goal without calling it peering. Putting it in writing before discussing makes it a more cumbersome and formal process. Organizations need time to think about and discuss requests with their colleagues. Many questions need to be weighed. Creating a conversation rather than a demand is going to be the first step towards success.

Agreement Terms

Agreement terms are variable. Ask for a term (length of agreement) that is reasonable. There are public references setting baselines. Google and Level(3) signed a multiyear^fb agreement (Level3)^a. The FCC opened Charters network for seven years (FCC, 63, 132). Charter files all interconnection agreements (FCC, 65, b.135). Read the filings and see how you compare or what you may be able to glean for your own baselines.

Making Interconnection Decisions

Know where you need capacity and where you don’t. In the case where capacity no longer makes sense, be sure to consider falling back to another connection. In all cases use sound business judgement. Pushing and pulling the levers of an interconnection agreement should help you to operate your network more efficiently. Before making rash utilization judgements, understand the value of an underutilized link. Will that less than 1 gb/s of traffic now be hauled over a backbone? Are there premium services contained in that link that will suffer performance wise and trigger an SLA with a customer? You may consider some rules of thumb when making such decisions. 

Traffic Risk Reference Table

Set up your own reference table. Establish how much used and available capacity you have. Once you figure that out, create step levels of risk using spaced percentage points. For example, 1%, 5%, 10%, 20% and so on.

 Table 1

Traffic Cost Reference Table (Cost assumed at USD$2.00 per mb/s, effective cost)

Once you build your own reference table, also map costs to it. You can scale your impacted capacity against your average cost and come up with a financial impact. Use both to understand risks and costs related to network decision making. This should also include potential income.

  Table 2

Summary

The idea that all interconnection is a wink and a handshake is a myth. While that approach may work for setting up and tearing down BGP sessions on an IXP, it does not work to protect critical business interests. Operating a network at scale is hard. Removing as much risk as possible is key. Using balanced, well thought out, interconnection agreements to reduce that risk should be a priority.

Other considerations

• Good relationships play a part in interconnection. Cultivate and maintain them, they are an asset. You will use them time and time again. They will become key to your career. It is true that the squeaky wheel gets the grease, but that only works once. Be a good partner, behave like a respected network enabler. You will get extended mileage from your interconnection reputation time and time again.

• The Internet has changed, and keeps changing. So has the relevancy of the methods and procedures of the past. Denser peering and interconnection is the answer to less back-haul, and more billable traffic to downstreams or less expense is one of many underscores to mutual benefit. Admittedly, some will never feel that mutual benefit is established without parity measured by a list of check-boxes. When there is mutual benefit, not much else matters other than exploiting that to the fullest potential. In many networks, interconnection remains a culture from 2001. Education will help some. Pointing to the change in traffic movement including rapid internet bypass uptake may awaken others.

• The days of describing networks by “tier” are over. It was an indicator of stature for network operators. The traffic patterns of the last ten years have dramatically shifted. It is true that technical IP networking is still the same, but nothing else is. The rise of the cloud and CDNs as fast, best performing, bypass or on and off ramps to the global Internet have impacted flows. In spite of a widely held belief that the middle layer is king, this is no longer the case for most. Cloud and CDN are successfully bypassing many directly to the eyeballs including SFI with the enterprise. They are directly connecting through settlement free peering regardless of stature.

• Ask peers to abide to the Routing Manifesto http://www.routingmanifesto.org/manrs/. We have too many security incidents that could be prevented. MANRS seeks to improve operations and reduce incidents. Read it, read it again and try to codify its relevancy in your agreement.

• Conduct regular reviews of peering policies. The peering policy forms the basis on which the agreements are based. Reviews are recommended to ensure the policy and the business objectives still align. It also helps to maintain a regular cadence of mapping policy to changes in the Internet, the traffic flows, the law and culture. Do not be afraid to push back. As long as you have the proper data points and the context. Understand how they apply to the relationship. You won’t cause damage.

• Coordinate with your internal teams. Check with your account and sales teams to make sure you understand the relationship with a peer. What crossover spending do you have? Are you  asking for a benefit that is worth $1 and the other party buys $100 worth of your product? It is prudent and savvy to be sensitive to balance of trade. When the sales team gets rankled, you’ll hear about it. Your partner will also use it as a technique to avoid interconnecting or to push fees in your direction. Carefully review balance of trade and your strategy.

• Challenge and respond. If you have strong arguments, you will have confidence in them. Some parties may try and weaken your arguments by denying them. That can be effective. It can create a logjam that invites doubt. It also invites second guessing. Have confidence that you are correct. Be careful not to confuse confidence with arrogance. Be open minded to being wrong, but keep your eye on the ball at all times. You aren’t likely to need to surrender a point entirely unless it makes sense. You may need to propose modifications as a sound counter-argument.

The internet, interconnection and peering is a business. Act accordingly.

References

1. Level 3 “Level 3 and Google Reach Settlement-Free Interconnection Agreement”, Jan 15, 2016. Accessed October 18th, 2017. http://investors.level3.com/investor-relations/press-releases/press-release-details/2016/Level-3-and-Google-Reach-Settlement-Free-Interconnection-Agreement/default.aspx
2. Federal Communications Commission “FCC 16-59”, May 10, 2016. Accessed August 22, 2017 http://www.citationmachine.net/bibliographies/236587290?new=true
3. Comcast “Comcast Peering Policy” October 2013. Accessed August 23, 2017. https://www.xfinity.com/peering
4. Charter “Charter Peering Policy”, Unknown, Accessed August 24, 2017.  https://www.spectrum.com/policies/peering
5. Cogent “Cogent Peering Policy”, Unknown, Accessed August 24, 2017. http://www.cogentco.com/en/13-network/peering
6. Brander, S. “BCP 14”, March 1997, Accessed August 29, 2017. https://www.ietf.org/rfc/rfc2119.txt
7. Brodkin, J. “Why YouTube Buffers”, July 28, 2013. Accessed October 3, 2017. https://arstechnica.com/information-technology/2013/07/why-youtube-buffers-the-secret-deals-that-make-and-break-online-video/2/
8. Geoffrey, J. “The Tao of Programming”, 1987. Retrieved October 12, 2017
9. Lodhi, A., Laoutaris, N., Dhamdhere, A., Dovrolis, C. “Complexities in Internet Peering: Understanding the “Black” in the “Black Art””, In Proceedings of IEEE Infocom 2015. Accessed October 10, 2017. https://www.caida.org/~amogh/papers/peeringcomplexity-INFOCOM15.pdf
10. Steenbergen, R. "The Folly of Peering Ratios". Accessed October 16, 2017. https://www.slideshare.net/RichardSteenbergen/a-guide-to-peering-on-the-internet
    

Editor's note (18-Oct-2017)

a) A suggestion was made to cite a first hand source. A citation reference to lightreading.com was replaced with a reference to a press release from Level 3, detailing Level 3's interconnection agreement with Google.

b) Initially a ten year contract term was mentioned in this article. While there is one public source suggesting a 10 year term, there is no second source to support the assertion. The phrase "ten-year" was replaced with "multi-year" to align with information made available to investors.

Acknowledgements

The Internet operates best when there is cooperation and collaboration. We thank Patrick W. Gilmore, Arnold Nipper and Christian Kaufmann for their time and efforts. They provided feedback and helped improve this article.

Contact the Authors 

Martin Hannigan
Boston, MA, USA
hannigan@gmail.com
+1 617 821 6079

Job Snijders
Amsterdam, NL
job@instituut.net
+31 6 54942365