How Far, How Fast?

The chief source of friction is the question of how fast the transition to Internet TV will take place and who will pay for it. Network operators are used to upgrading capacity in the course of routine maintenance, as older equipment can generally be replaced with much higher capacity equipment at no increase in cost.More rapid upgrades require increased investment, which only comes about with a plausible business plan. This is not a situation that cries out for regulatory intervention, however. The principal needs are for capital and innovation; aggressive regulatory action simply complicates these problems.

A year ago, I wrote a guest blog for GigaOm predicting that the vague anti-discrimination rule in the FCC’s proposed net neutrality framework could easily become entangled in interconnection disputes between large network operators despite the agency’s adamant insistence that they had no such intention. Last week the chief of the FCC’s Wireline Competition Bureau called Comcast practically as soon as the story broke, apparently demanding details, despite the fact that we don’t have an actual net neutrality law in the United States at the moment. Network operators didn’t want to think about the implications of net neutrality for their highly unregulated business a year ago, and told me so in 140 comments to my GigaOm post. They’re now aware that denial isn’t a constructive strategy and have started to pay attention.

As I explained in that post, the rise of video streaming services such as Netflix Watch Instantly is changing the Internet. Sandvine’s Fall 2010 Global Internet Phenomena Report confirms the scope of the change:

Within North America, we observed that Real-Time Entertainment is the largest contributor to data consumption on both fixed (43% of peak period traffic) and mobile access (41%) networks. Within that category, Netflix is a major source of content, representing more than 20% of downstream traffic during peak hours on fixed access networks, and is heaviest from 8pm to 10pm.

This shift toward Internet-delivered TV hasn’t yet been accompanied by significant cord-cutting on the part of cable TV customers; subscriber counts are down, but only slightly. Internet streaming has only just started to take off; by one estimate, fewer than two percent of Netflix customers are responsible for the company’s 20 percent share of the prime-time traffic load. But Internet operators have already taken some very significant steps to ensure that the Internet doesn’t collapse under the massive new load that’s just around the corner: Content Delivery Networks such as Akamai and Limelight Networks have installed video servers in colocation centers and Internet Exchanges (IXs) as close as possible to the ISP networks that carry bits to the last mile, where the users are.

These servers attach to ISP networks through a few feet of cable, not across the long-haul links that have been the source of Level 3’s traditional revenue stream. (They make use of long-haul links, but only to seed their servers with movies that will typically be downloaded hundreds or thousands of times without any further perturbation of the long-haul network.) Colos, as they’re called, exist to make this kind of interconnection fast and cheap, and succeed because distance drives cost in network economics. ISPs install routers in as many colos and IXs as possible, the better to keep their costs low and their performance high, so all the CDNs need to do to reach them is connect through a common Ethernet switch in the colo center.

Money Follows Packets

The fact that Internet businesses have to share common facilities in order to exchange IP packets doesn’t have much effect on the Internet’s economic agreements, however. No pair of networks exchanges packets without an agreement, either a bilateral one between the two networks directly or an indirect one between a third network who connects to both. These agreements are as varied as the networks themselves, but they generally assign value to the interconnection based on the share of the work that the networks do to move packets from one Internet end point (user or server) to another. When some Network A connects to a Network B of similar size, scope, utilization, and capacity, they’ll typically interconnect with no money changing hands; this is traditional Settlement-Free Interconnection (SFI) or peering. But if Network A is a small regional network and Network B has international scope, undersea cables, and massive redundancy for quality and reliability, Network A will pay Network B a volume-based transit fee for moving its packets. When two networks have connected as peers for some time but changed circumstances alter the equal division of labor, they de-peer, sometimes acrimoniously, because de-peering affects the profitability of networks that have to pay more transit fees to compensate for their lack of peers.

There is a long history of noisy de-peering disputes between network operators. Wikipedia lists several of the best known ones:

· BBN Planet vs. Exodus Communications

· PSINet vs. Cable & Wireless

· AOL Transit Data Network (ATDN) vs. Cogent Communications

· Teleglobe vs. Cogent Communications

· France Telecom vs. Cogent Communications

· France Telecom (Wanadoo) vs. Proxad (Free)

· Level 3 Communications vs. XO Communications

· Level 3 Communications vs. Cogent Communications

· Telecom/Telefonica/Impsat/Prima vs. CABASE (Argentina)

· Cogent Communications vs. TeliaSonera

· Sprint-Nextel vs. Cogent Communications

The traditional way that operators have resolved questions about payment for interconnection is to measure traffic between the two networks and to have the money follow the direction of the dominant traffic flow. If Network A sends more traffic to Network B than vice versa, Network A pays. This works for the networks that have historically served as backbones, carrying traffic for others but not generating any of their own. The “sender pays” model works well enough to capture the economics of long-haul networking even though it doesn’t single out all the cost factors; ensuring that money flows in the same direction as traffic has been a good-enough simplifying assumption to keep the backbone ecosystem healthy and competitive.

Figure 1: Money follows traffic on the Internet

Comcast points out that Level 3 used the “money follows traffic” argument in its dispute with Cogent Communications; this is a pretty good argument, as this formula has been definitive in past de-peering disputes. But Level 3 argues that the current dispute is different from the Cogent incident because Comcast is an ISP rather than a transit network, so the traffic flow doesn’t matter; they say they’re simply delivering packets that Comcast’s customers have requested, so there’s no reason money should change hands. They’ve asked Comcast to open up 30 new ports for their traffic, with a capacity of 300 Gigabits per second or more, for no charge.

This is a clever argument, but it’s not persuasive. Level 3 is in this dispute because they’ve stepped outside their traditional role as a transit network and become a value-added content distributor, effectively a proxy for Netflix. It makes no more sense for Level 3 to demand free transit from Comcast than it would for Netflix to demand free service from Level 3; after all, Level 3 only serves up movies because somebody asked for them. While Level 3 apparently undercut Netflix’ former dominant CDN, Akamai, to win this business, Level 3 is still charging Netflix to deliver movies to the Netflix customers who’ve requested them; just as Hollywood is charging Netflix for the movies in the first place. In fact every packet (except the malicious ones) on the Internet flows because someone asked for it – that fact alone doesn’t entitle anyone to a free ride.

Aligning Incentives

The policy question about disputes like this one is how to align the payments between all the parties – Hollywood, Netflix, Level 3, Comcast, and Internet users (remember us?) so that investment, innovation, good behavior, and the social good are maximized. We want companies to invest, take risks, and do good things, and we recognize that well-functioning markets are the best means of aligning incentives. Level 3 argues that the market has failed to function properly in this instance because it can only reach Comcast’s customers through the Comcast network, which they can’t access without paying a fee they regard as unreasonable. Hence, Level 3 has complained to the FCC and wrapped itself in the net neutrality flag. The carrier apparently feels safe in adopting this tactic because the FCC has declared that the net neutrality rules are only for ISPs, not for content owners or distributors; so Comcast has to play by the rules, but Level 3 does not.

In fact, the FCC certainly has insisted that net neutrality doesn’t apply to transit networks and peering disputes, and the present dispute shows that their position is no more tenable than the former regulatory status quo that classified Internet use via DSL as a regulated Title II service while cable modem was a regulation-exempt Title I affair. Level 3 insists that no Internet peer of an ISP network can ever accept as many packets from the ISP as it transmits; the wild claims of Web 2.0 advocates notwithstanding, ISP customers download 5 to 10 times as much, if not more, than they upload, and the only Internet applications that generate symmetric traffic are interpersonal communications such as VoIP and video calls and P2P file sharing.

While asymmetry is undeniably true, it’s hardly a new phenomenon. In the days before CDNs, Level 3 would have charged Netflix a hefty fee for transporting its packets from a Netflix server to the Internet as whole, spending a portion of the fee on networks that provided it with transit to some far corners of the country, and nobody complained about that. In the current scenario, it’s a safe bet that Level 3 is paying some networks for transit today, and not just to Comcast or the other ISPs. So the fact that Level 3 and Netflix have cut their costs by locating Hollywood’s content in colocation centers hasn’t eliminated Comcast’s expenses; if anything, it’s increased them by presenting greater traffic volume to Comcast’s network.

There’s a larger issue behind the dispute that the emphasis on packet volumes and the foot-stamping about net neutrality threatens to obscure, however. Even in the pre-CDN days where peer networks were all of one kind, packet volumes alone didn’t capture the full panoply of network economics. Network costs are determined not only by how many packets a network carries, they’re heavily shaped by the distance the packets must be carried; long pipes cost more to build and operate than short ones. This is why networks such as Level 3 and its competitor Global Crossing have always cultivated the image of the “planetary supplier” who could reach everyone, everywhere, all the time; that’s the rationale for the first part of Global Crossing’s name.

So even in the case where two networks hand off equal numbers of packets to each other, the network that carries them furthest has higher costs and is entitled to a larger share of the fees, all else being equal. This discrepancy has to be accounted for in any rational peering or transit agreement, as those that are public all do. This idea also serves the public policy goal in which rewards flow with investment; the more extensive the infrastructure, the greater the fees it should generate.

It’s from this pillar of network economics that Level 3 takes the biggest tumble. While the people who run Level 3-style transit networks from windowless basements full of massive screens and racks upon racks of network routers cooled by shrieking fans like to fancy themselves as the wizards of the Internet, their networks are declining in importance compared to the broadband networks the ISPs operate, according to Arbor Networks’ landmark study of Internet interconnection. While the Top 10 networks in terms of traffic in 2007 were all transit providers, two networks have entered the Top 10 as of 2010 that are not traditional transit networks, Google and Comcast. The Internet’s edge is thickening, and its core is diminishing.

Network Coverage

Another way of seeing this transition is to examine network coverage. The term that measures a network’s extent is “route miles,” which is simply the miles of cable it incorporates. Level 3’s latest annual report lists them as a major asset:

• approximately 79,000 intercity route miles in North America and Europe, which we expect to ultimately reduce to approximately 54,000 intercity route miles, connecting 22 countries; and

• approximately 125 markets having metropolitan fiber networks containing approximately 27,000 route miles in the United States and Europe.

Packets times route miles roughly equals the value of the network to paying customers, and it sounds like Level 3 has a lot, even if those intercity route miles now on the decline are cheaper to build than the ones in populated areas.

Level 3’s network is dwarfed by Comcast’s however:

The Comcast network has seven times the route miles of Level 3’s network, and the gap is increasing as Comcast’s network grows and Level 3’s shrinks. So even if Level 3 happened to be using its entire network to serve the joint customers of Netflix and Comcast, it wouldn’t be doing a fair share of the work. When we include the fact that their CDNs are actually moving most of the Netflix packets only a few feet before dropping them into Comcast’s routers, it should be abundantly clear that regardless of what Level 3 is charging Netflix, they’re not doing as much of the work as Comcast.

Regardless of whether Comcast has a termination monopoly to its current customers and regardless of whether it sells a service that competes with Netflix, as the carrier doing most of the work, they’re entitled to reimbursement for a portion of their costs, and they should be able to do so in some rough proportion to the share of the work they’re doing – in both investment and operations – to make Netflix streaming a satisfactory experience. Of course, they do capture a great deal of the value from the subscription fees they collect from their customers, but most of them aren’t using Netflix streaming yet, so in effect those who aren’t streaming are subsidizing the cost of Netflix’ heavy traffic load within Comcast’s extensive network.

As more users enjoy the Netflix service, a big problem is looming for the ISPs which they can’t resolve as cheaply as the CDNs have resolved the Internet’s long-haul capacity problem: When Netflix penetration climbs to 10 percent of prime time usage, cable company networks will reach saturation in their second mile, just above the DOCSIS M-CMTS that forms their fiber-based head ends and below their IX connections.

Figure 2: Internal Structure of the Cable Network (Source: Video over DOCSIS, Michael Cookish.)

The other way to look at this is to take Level 3 out of the picture altogether: The real battle here is between Netflix and the broadband ISPs.

Rent vs. Own

The Netflix business model is the opposite of the ISPs’: Where ISPs are locked into a never-ending cycle of investment, upgrade, and replacement on their tangible facilities, Netflix relies on rented facilities for everything from IT to distribution. To maintain Quality of Service, they encode and store Hollywood’s movies at multiple levels of compression, sending most them out at the equivalent of standard definition and others closer to HD quality. They rely on Level 3 and Limelight for streaming, and still generate most of their revenues by mailing DVDs out the old-fashioned way. They buy a lot of DVDs from Hollywood, and they use their position as the world’s largest purchaser of movies on plasticware to negotiate attractive rates for streaming the less attractive titles across the Internet. (Netflix rates movies on a five-star scale, and you’ll be hard-pressed to find a month’s worth of four or five star titles available for streaming.)

Netflix is hoping to change this, and the key is to drive down their costs as low as possible to make more cash available for licensing attractive titles for streaming. That begins by shifting customers over from DVDs to streaming by offering an $8/month streaming-only plan an addition to the original $20/month DVD plan. It then focuses on driving the costs of streaming down as low as they can go; they’re already much lower than postage and handling for DVDs. Lower costs for Netflix translate into lower costs for the consumer, which has the effect of stimulating more demand for streaming video. Against this scenario, capacity limits or “bit caps” for consumers are looming as well.

Making the claim that Comcast is violating net neutrality is probably best understood as a business strategy for Netflix, a tool that may buy leverage with Hollywood and the ISPs. So whatever else the dispute between Comcast and Netflix/Level-3 may be about, it’s less a matter of principle than a business issue. Unlike most of his entrepreneurial peers in Silicon Valley, Netflix CEO Reed Hastings is not a political babe in the woods; he was appointed President of the California State Board of Education by Democratic governor Gray Davis, and re-appointed by Republican governor Schwarzenegger only to lose a bitter re-confirmation fight. He’s also a long-time charter schools activist and a bankroller of gubernatorial candidates and ballot initiatives. The value of friendly regulation isn’t lost on Hastings.

Netflix has made a business calculation and determined that the stars are aligned such that playing the net neutrality card now will provoke the most favorable reaction from Comcast and the FCC: The agency is struggling to assert net neutrality rules against the objections of a hostile Congress, Comcast wants the FCC to approve its merger with NBC Universal, and Level 3 needs to succeed in the CDN business because the transit business is declining. Regulators should not buy the claim that paying for transit from a few widely separated colos to ISP end users distributed across three-quarters of a million route miles is unfair.

The economics of shared, in-place network facilities depend on utilization. While consumers may feel entitled to use 100% of the capacity of a service that was sold to them on an “up-to” basis of X Mbps, this isn’t a realistic expectation. Consumer broadband can only be provided at low cost as long as we don’t all don’t use the entire “up-to” capacity all the time; business class services that provide guaranteed bandwidth are typically 3 to 10 times as expensive as consumer grade services for good reasons. Networks are designed according to assumptions about utilization, and cable modem networks are already far beyond their original parameters. These networks can be upgraded by Modular CMTS bypass, node splits and virtual nodes, and more capacity can be appropriated for Internet access from broadcast TV as switched digital video is adopted, but regulatory and financial barriers make it impractical for operators to take all of these steps overnight. One means of slowing the transition to unicast TV is to enforce more restrictive usage limits on users; the day of “all you can eat” pricing is necessarily coming to a rapid end.

In addition to the architectural assumptions embedded deep inside the cable system design, cable modem networks have physical distance issues that make upgrading end-to-end capacity more expensive than it is for IP- and Ethernet-based colo center switches. The cable modem network was ultimately designed around different video formats and distribution models than those employed by Internet TV. Internet video takes a very circuitous path within each cable network for historical reasons. Cable needs to gravitate toward more open, standards-based solutions, just as the mobile network is doing with LTE, but it also needs a clear migration path to this nirvana that hasn’t been spelled out yet, and it requires profits from current operations to pay for a massive upgrade. The Internet standards for routing, security, and video streaming are also not what they should be, and Hollywood has been slow to embrace the on-line distribution model.

The Pressure of History

These circumstances don’t represent bad faith on anyone’s part; they’re merely the result of imperfect planning and technology forecasting reacting to a new way of doing things that has just become practical for a small portion of the Internet population. Growing Internet video streaming from a boutique business to a mainstream one is the challenge, and it implicates essentially all the players and business models in the Internet ecosystem. Netflix is playing the role of the provocateur at the moment, probably realizing that they only have a business as long as Hollywood doesn’t enter the Internet video game on its own behalf in a big way.

Netflix is the kind of business that appears to have been designed for acquisition by a larger player, like Hastings’ last business, Pure Software. Its astronomical price/earnings ratio (70, vs. Apple’s 20, for example) makes it a tough candidate for auction in this economy so they’re pressing to become profitable enough to justify their share price and to get out ahead of Hollywood in streaming. Netflix is seeking to expand its on-line content portfolio substantially:

Netflix is making an aggressive play for in-season episodes of hit TV shows to expand its Web streaming service.

The company is in talks with studios about gaining access to current episodes of primetime shows and is willing to pay between $70,000 and $100,000 per episode, according to a person familiar with the matter. Netflix had no comment.

If this move is successful, it will raise cable companies’ costs and reduce their revenues from current operations, forcing end-user price increases. Costs will increase from the requirement for an accelerated transition from the current generation of Modular DOCSIS CMTS’s to Integrated CMTSs with native IP support. It will also accelerate cord-cutting, reducing cable revenues, so adjustments to the cable modem revenue model are inevitable if for no other purpose than to keep cable modem networks alive after the shift to Internet TV. At the end of the transition from cable companies’ current system, optimized for broadcast TV, to a system optimized for unicast TV over IP, consumers will have more choice, but will inevitably be forced to pay higher bills simply because unicast TV is millions of times less efficient to deliver than broadcast TV is. A slower transition would be less costly to end users and cable companies alike as advances in technology would allow increased capability in the course of upgrading retired equipment. Managing the rate of transition from broadcast TV to Internet TV appears to be sound policy.

Figure 3: Bypassing DOCSIS with Native IP Tunneling (source: Video over DOCSIS.)

Comcast has a standing offer of a transit service tailored to firms like Level 3 and Netflix: They offer “paid peering” at a lower price than standard Internet transit to those who want on-net access, or direct access to Comcast customers for large volumes of data. Paid peering is unlike transit in that it doesn’t allow users to employ the Comcast network to reach a third party. It’s essentially peering for a fee, estimated at a third to a half the price of transit and possibly much less.

Pair peering is nothing new; AOL’s small but aggressive peering department offered it years ago, and Level 3 offers a variation called “on-net access” to its commercial customers. Paid peering is a capacity-based surcharge that can be combined, in principle, with a Quality of Service commitment that would allow more Netflix customers to enjoy HD-level viewing as the company makes it more available. The paid peering surcharge also enables the cable companies to pay for the network upgrades they need to make to increase their capacities by the ratios of 100-1000 times that will be necessary to support universal Internet TV when all the tricks are employed. It extends the value chain for Internet TV all the way to the essential infrastructure that makes it possible. The alternative is much higher volume-sensitive pricing to the consumer that can only have the effect of keeping people in the broadcast fold and away from Internet TV out of fear of outrageous, unpredictable bills. While volume caps for the consumer are necessary, some cost sharing is also in order. In the end-to-end architecture of the Internet, video servers are just as much network end points as consumer equipment are, and ultimately must to conform to the same economics.

Conclusion

This dispute started with Level 3 asking Comcast to make 30 ports (a combined capacity of 300 Gigabits/second at a number of different locations) of access bandwidth available for free use. There’s nothing wrong with the request for new access ports as long as a portion of the money Level 3 bills Netflix for their use follows the traffic. That’s the established convention, and it’s reasonable. The firms are capable of working out this dispute on their own; other CDNs have been able to reach satisfactory terms with Comcast and the other ISPs, and Level 3’s not doing much different from what Akamai and Limelight have done in the past. The FCC would do well to step back and let the firms work out the terms of an agreement among themselves.