The controversy over the deal between Comcast and Netflix for paid access to the Comcast residential network is heating up, and that’s just inside Netflix. Immediately after the two firms agreed on a business relationship that gave the Netflix Content Delivery Network special access to the Comcast network, Netflix CEO Reed Hastings said he was happy with the deal, but now that he sees Netflix has indeed secured more reliable delivery of its TV programming from the deal, he’s unhappy. This is a curious situation.

Happy Netflix issued an upbeat joint statement with Comcast upon the consummation of the deal in late February:

Comcast and Netflix today announced a mutually beneficial interconnection agreement that will provide Comcast’s U.S. broadband customers with a high-quality Netflix video experience for years to come. Working collaboratively over many months, the companies have established a more direct connection between Netflix and Comcast, similar to other networks, that’s already delivering an even better user experience to consumers, while also allowing for future growth in Netflix traffic. Netflix receives no preferential network treatment under the multi-year agreement, terms of which are not being disclosed.

Less than a month later, an unhappy Netflix wrote a blog post expressing apparent dissatisfaction the deal:

[On some] big ISPs, due to a lack of sufficient interconnectivity, Netflix performance has been constrained, subjecting consumers who pay a lot of money for high-speed Internet to high buffering rates, long wait times and poor video quality. A recent Wall Street Journal article chronicled this degradation using our public data. Once Netflix agrees to pay the ISP interconnection fees, however, sufficient capacity is made available and high quality service for consumers is restored.

To be clear, Netflix isn’t upset about high service quality for its customers, it’s upset about having to pay ISPs for the kind of interconnection that assures this high quality of service. Netflix prefers to locate servers on ISP premises that bypass standard forms of ISP interconnection, but so far very few ISPs have been willing to grant them this access; they prefer for Netflix to connect to their networks the same ways that other services do, at Internet Exchange Points and collocation centers, and under transit and/or peering agreements.

How the Internet Works

The technologies for Internet interconnection aren’t terribly complicated, but most of the time there’s very little interest in understanding them. What I’d like to do here is explain the options so readers can see where the current Netflix/Comcast interconnect system falls on the spectrum and where the alternate form of interconnection proposed by Angry Netflix would fall. Understanding the options will make it easier to judge whether the current Netflix demand is reasonable. By way of background, you might want to check out a blog post I wrote for GigaOm in 2009, How Video is Changing the Internet, and a short paper I wrote on the advent of the dispute between Comcast and Netflix in 2010, Now Playing: Video over the Internet.

The issues haven’t changed; in the blog post, I speculated that the FCC might feel forced to include interconnection regulations in its net neutrality rules, and yesterday net neutrality maven Susan Crawford applied just that kind of pressure:

…regulators need to intervene [in the interconnection space.] U.S. interconnection markets are at the moment perfectly engineered to raise revenue for Comcast and AT&T…

It’s not just Crawford: I’ve been told by certain FCC employees that the cost of interconnection should be zero. So what is interconnection, how does it work, and how is it financed?

Interconnection is how the discrete, privately-owned networks that comprise the Internet can talk to each other. While the Internet looks and feels to the user like one big, unified network, in reality it’s a collection of some 50,000 privately owned networks that all interconnect to each other, one way or another. The most common way for an individual network such as the Comcast network or the Netflix network to interconnect is through a deal with an interconnection specialist; in the Internet industry these are known as “transit networks”.

Small ISPs in particular have contracts with transit networks such as Level 3 who do two closely related things: They span the globe, either directly or indirectly, and have permission from all the other networks, either directly or indirectly, to interconnect and to exchange traffic. Level 3, for example, has a global network of some 100,000 route miles of fiber optic cable, most of it in North America and Europe. It has deals with other global networks for the reciprocal exchange of traffic to and from the countries where it doesn’t have its own network. In addition to its own network, Level 3 has deals for interconnection with all the networks its network touches. Level 3’s transit customers pay it for carrying their traffic to and from the rest of the Internet, which includes exchanging it with ISP networks around the world.

So interconnection is how networks communicate with networks. It’s fashionable to say that most interconnection deals are settlement-free, meaning that no money changes hands to make them happen. Crawford mentions a study commissioned by the OECD estimating settlement-free peering (SFP) is the norm, applying in 99.5% of cases. This study is anecdotal and a bit misleading, because networks that make SFP deals are actually trading services of equal value with each other; it’s essentially the same as A charging B a million dollars while B charges A a million dollars, so no actual money needs to change hands (and no taxes need to be paid.) When a wireless ISP in Mineral Wells, Texas contracts with Level 3 for transit, the Texas firm doesn’t have a service of equal value to trade with Level 3, so it pays cash.

In between SFP and transit, there are several variations of service and price. One hybrid is partial transit, in which a network with some SFP agreements will contract with a transit provider to reach the remaining networks with which it doesn’t peer. There is also something called “on-net access” or “on-net routing” that’s a hybrid of carriage and partial transit; Level 3 provides so-called “on-net services” in which they extend a cable from their existing network to a customer site outside their current service area, effectively expanding their service area to capture a new user’s business.

The variation we’re interested in today is called “paid peering,” which is a more or less direct connection between two networks that do not have services of equal value to trade, but where a service as extensive as the typical transit service isn’t required. The Netflix/Comcast scenario is an example of paid peering. Netflix places equipment in ten or so specific Internet Exchanges, as does Comcast. Comcast also makes fiber optic cables and/or light frequencies (“lambdas”) available between its residential network and the ten IXs, and Netflix makes content available to travel on Comcast’s lines. Both pay the IX for ports on an Ethernet switch, both advertise routes to each other, and they’re off and running. We don’t know how much money changes hands in this particular deal, but informed sources say it’s not much, most likely less than Netflix was paying it transit providers for a smaller amount of capacity before making a deal with Comcast.

There are several reasons why paid peering deals – which have been around since the 1990s – are becoming commonplace. For one thing, they help to ensure stability in the operation of ISP networks. Before Netflix built its own CDN, it was in the habit of changing CDNs every six months according to which offered the best price at any given time. Initially, Netflix probably used Akamai – most people do – but along the way, they shifted all or part of their considerable business to Level 3, Limelight, and others. Each new contract meant that 30% of the prime-time traffic on the North American Internet suddenly began to appear at a different exchange without any warning. This practice caused the ISPs no end of grief, because suddenly seeing a third of the Internet’s total traffic in a new location one fine Monday morning is pretty close to the definition of a denial of service attack.

Initially, they reconfigured their networks to match the load, even though that meant that circuits they bought or leased to handle the traffic load of the last configuration were no longer needed. At some point, apparently, the major ISPs stopped responding to Netflix reconfigurations so suddenly and simply left it up to Netflix and their CDN du jour to present their traffic where they already had the capacity in place to deal with it.

There’s little question that the major ISPs have always had the total capacity somewhere in the interconnection fabric to handle the Netflix load. There has been a mismatch in many instances between the desire of Netflix to unload traffic at particular locations and the ability of Comcast and others to handle it at those particular locations. The contract presumably specifies where the two networks will interconnect and how they will notify each other of changes in their topography. So that’s one good reason to have a formal arrangement, and there are others related to price; Comcast can sell interconnection to Netflix cheaper than Cogent or Level 3 can by eliminating the middle man, even if Netflix remains a middle man between the Comcast customer and the Hollywood studios who produce 99% of the programs Netflix sells.

But what about this equal value thing, where does that come from and what does it mean? The Internet practices something called “hot potato routing” where Network A dumps off traffic to Network B at  the earliest opportunity, and Network B behaves similarly. So a customer of Network A in Los Angeles who is sending an email to a customer of Network B in New York will have his message carried from LA to New York over Network B’s lines and his peer will have his messages carried in the opposite direction over Network A’s lines. The networks are performing tasks of equal value (and equal cost) for each other, so they can barter with each other for the simple carriage of bits. Netflix isn’t in a position to barter carriage since they just don’t do it.

The bottom line is that all forms of interconnection have a price attached, either explicitly or implicitly, and those who can’t barter for services of equal value pay cash. This is how the Internet has worked as long as it has been a commercial service. Before it was commercial, the universities who took part in the Internet paid for connections to the National Science Foundation backbone.

Because Netflix is the largest producer of traffic in North America, it can negotiate carriage of its bits on a wholesale basis and secure a lower rate than you and I pay for the ability to interconnect our home networks to the other networks that comprise the Internet. That’s perfectly consistent with historical practice as well.

Another dimension of interconnection is purely economic: the carriage agreements that bring packets to IXs are typically written around volume-based prices and assured (more or less) levels of service. This is directly contrary to what’s allowed by the FCC’s various attempts to write net neutrality rules banning quality-based service agreements. Because the disconnect between classical Service Level Agreements for bit carriage and the vision of net neutrality is so large – these are really two different worlds – net neutrality advocates have generally insisted that net neutrality doesn’t apply to interconnection agreements.

We all pay to “connect to the Internet”, which is to say we all pay to interconnect with the thousands of networks that comprise the Internet. Some pay lower prices per bit than others, and those who pay the lowest prices – as Netflix does – also send the greatest volume of bits. This is not terribly surprising, and it’s certainly not unprecedented.