How Interconnection Works

iStock_000034051058_LargeInterconnection played a big role in the FCC’s 2014 Open Internet rules. The FCC rules give the FCC the power to examine interconnection agreements and to change them if it wishes, which is correctly seen as a reaction to the interconnection problems Netflix had with the major ISPs in early 2014. Oddly, the FCC admits it lacks expertise in this area:

  • 31. While we have more than a decade’s worth of experience with last-mile practices, we lack a similar depth of background in the Internet traffic exchange context. Thus, we find that the best approach is to watch, learn, and act as required, but not intervene now, especially not with prescriptive rules. This Order—for the first time—provides authority to consider claims involving interconnection, a process that is sure to bring greater understanding to the Commission.

So the FCC is regulating interconnection in order to understand it.

Interconnection is the means by which each of the 40,000 networks that comprise the Internet communicates with the others. When I send you an email, my ISP can deliver it to your ISP because these two ISP networks interconnect. Ultimately, they interconnect through one or more Ethernet switches located in Internet Exchanges or similar private facilities in a dozen major US cities. In the abstract, these switches are similar to the Ethernet switch that’s built-in to your home router; in practice, carrier Ethernet switches are bigger and faster.

So my ISP and your ISP lease ports on one or more Ethernet switches and voila, they interconnect. OK, that’s only part of the story, the easy geeky part. There’s also a business part, and it’s more complicated. Not only do our ISPs need a common switch, they also need an agreement to interconnect. These agreements are of two general kinds, peering and transit. A peering agreement means our ISPs don’t exchange any money, only packets. But a transit agreement means that one of the network owners pays the other owner for carrying their traffic.

The main reason for transit agreements is the fact that not all of these 40,000 networks are physically capable of sharing switches with each of the others; a small ISP in the boondocks can’t use Ethernet to connect to a small ISP in Kirgizstan, for example. So the small ISPs pay specialized transit networks (Level 3, Tata, Cogent, et al.) to interconnect them with the rest of the Internet. The transit networks typically peer with other for free and charge everyone else.

In effect, consumer ISPs do the same job for their residential customers that the large transit networks do for theirs: they carry traffic from our Ethernet switches to exchanges where they’re able to interconnect with the rest of the Internet through their own negotiated agreements. (Residential ISPs do a lot more than this, but this is certainly one job they do.)

So this is pretty straightforward: networks that can interconnect directly do so and those that can’t simply pay a third party to interconnect them. But that’s the business side, and it’s not really that simple on the technical side.

Those that can peer are able to do so because they lease ports on Ethernet switches owned by third parties such as Equinix, the major owner of Internet exchanges in the US. Because of this, it’s actually accurate to say that everyone pays someone else for interconnection. In addition, the carriers who lease space in Equinix facilities pay for their own lines to connect their neighborhood networks to the Equinix facilities. These lines cost money, so the ISPs pay again.

So the idea that interconnection is typically free is an illusion; when and where this is the case, interconnection is an exchange of equal value, and that’s not really “free” in any meaningful sense. Level 3 and Tata have a settlement-free peering agreement because they do the same things for each other so no money needs to change hands. You can I pay our ISPs because they do things for us that we can’t do for them.

The interconnection disputes between Netflix and the ISPs raise a different set of questions. At a technical level, this interconnection doesn’t represent an exchange of equal value: the ISPs carry a massive traffic load for Netflix, and Netflix doesn’t carry any traffic for the ISPs. So tradition suggests that Netflix should pay. But tradition also suggests that Netflix should not pay, because the carriage the ISPs provide is within their own networks, and consumers are already paying for that. This is a fact.

But the ISPs collect interconnection fees from Netflix because Netflix represents a special case that isn’t well understood. For the past 15 years, Content Delivery Networks such as Akamai have paid fees to ISPs to cover favorable interconnection points and for pseudo-transit. Recall that typical interconnection takes place in a facility owned by a firm like Equinix and leased to the ISPs, large private networks, and transit networks. If the ISP provides the interconnection facility, it’s not unreasonable for a CDN to pay the ISP rent for access to these facilities, and they commonly do.

Similarly, if the CDN has dedicated facilities within the ISP network for high volumes of data, the ISP is effectively operating as a transit network, or a “pseudo-transit network” because it shares some facilities with the ISP’s customers. So this is the norm that Netflix falls within. Our statement at the beginning was that interconnection is either peering or transit, but there’s a third way for CDNs.

CDNs use optimized interconnections that increase speed and efficiency and actually reduces prices overall. Netflix used to lease CDN services from Akamai and Limelight, and now it operates a private CDN. Hence, it has to deal with ISPs the way that traditional CDNs do, with ongoing technical cooperation according to a contract.

This is only an outrage to the extent that the public doesn’t understand how CDNs and ISPs work together.