On Tuesday, March 11^th, Sprint Chairman Masa Son gave an interesting and exciting speech on the benefits of merging Sprint with T-Mobile that relied on some rather dramatic empirical claims. While the speech covered a lot of ground – it explained the reasons for leadership of the global economy from the 16^th Century to the present, for example – its essence was the claim that network users in the United States pay the highest prices for the lowest network speeds on the planet.  If this claim is true, our networking companies and our policy makers have some ‘splaining to do, as Ricky Ricardo used to say.

Son offered two sets of measurements, one each for wired and wireless networks.  Son has experience in the wired broadband sector in Japan, having been one of that nation’s first over-the-top unbundled ISPs back in the early 2000s, but he doesn’t propose to enter the wired broadband business in the US; rather, he promises to raise the speed of his wireless network to 200 Mbps, which would make it a credible competitor to wired broadband networks provided its price is low and its usage limit is high enough to allow users reasonable flexibility. This is his wired network slide:

Son claims that America’s LTE networks are the 15^th fastest in the world, out of 16 nations that have LTE networks; that would place us between Japan and the Philippines.  This is his global LTE network slide:

The source for this claim is OpenSignal, a company that makes a crowd-sourced smartphone app that it says has been downloaded 6 million times. I downloaded it myself to see how well it works because it seems unlikely that the US – the first country to deploy LTE at scale and current home about as many LTE users as the rest of the world combined – is falling behind the world in mobile networks, but it’s good to measure and keep an open mind.

Other sources don’t agree with the crowd-sourced OpenSignal. The Akamai State of the Internet report includes a mobile section developed in conjunction with Ericsson, one of the leading producers of wireless base stations. It doesn’t roll up data for entire nations, and it doesn’t identify carriers. It says the US has four wireless carriers with the following speed profiles:

ID            Q3/13 Avg         Q3/13 Peak
————————————-
US-2                6.4                   24.5
US-1                4.4                   14.0
US-4                8.4                   15.5
US-3                2.1                    6.3

Note that these figures are from all US mobile networks: 2G, 3G, HSPA+, and LTE, so we wouldn’t expect them to average as high as OpenSignal’s LTE-only figures. Ericsson doesn’t say which networks these are, but it’s reasonable to deduce that US-3 is Sprint, the company that consistently shows the lowest speeds in other network tests such as RootMetrics.

It’s also reasonable to guess that US-1 is T-Mobile on the basis of high peak and low average, since TM’s LTE is very limited but quite fast, while most of its network is spectrum-constrained and used by budget shoppers with older phones. That means US-2 and US-4 are AT&T and Verizon, and I wouldn’t hazard a guess as to which is which, and that doesn’t matter for the moment.

We can be reasonably sure that the LTE users cluster closer to the peak rates for all these networks than to the average, but I can’t say how close they are. The average of dominant carriers US-2 and US-4 is 7.4 Mbps for both LTE and non-LTE networks, which suggests that the LTE average for the nation would be a lot higher than the 6.5 that Son and OpenSignal represent as the “weighted average” of all LTE networks in the US.

This measurement discrepancy can come about from two different things. For one, the OpenSignal system only measures connections on phones that have downloaded the app, and this is likely to be something other than a random sample of all users. Crowd-sourced measurement is a lot like Internet polling, which is known to be inaccurate. If Internet polls elected presidents, the Howard Dean administration would have been followed by the Ron Paul presidency, after all. There’s no way to know for certain which way this bias goes;  I would guess that OpenSignal users tend to have newer smartphones than non-users, but can’t confirm it.

This leads to very significant factor: the performance of the device on which the OpenSignal application is run probably has more to do with its measurements than the network does. Readers may not realize that one of the most difficult problems in network performance measurement is the difficulty of separating the performance of networks from the performance of devices that use networks. Perhaps the best indicator of this is the performance figures Netflix publishes on American ISPs. According to Netflix, the fastest ISP in the US is Google Fiber, which is supposed to be a Gigabit network. But Netflix only sees 3.74 Mbps on Google Fiber; this is not because of limits in the network’s capacity, but rather because that’s all that the Netflix application asks of it.

After downloading the OpenSignal app to my Samsung Galaxy S II Skyrocket on the AT&T network, I measured a download speed of 12 Mbps and an upload speed of 3.1 Mbps. This is reasonably consistent with the speeds I see with Oookla Speedtest on the same device: 10 Mbps download, 7.8 Mbps upload, more or less. But there’s a catch.

When I use my Samsung phone as a wireless hotspot and run Speedtest from my MacBook Pro, I get figures of 21.01 Mbps download and 8.14 Mbps upload.

Test                         Download       Upload
———————————————-
OpenSignal LTE         12                    3.1
Ookla Phone              10.04              7.81
Ookla Hotspot            21.01              8.14

Now what does this mean? Did AT&T’s LTE network suddenly get twice as fast the moment I put my phone into hotspot mode and ran Speedtest from the Mac? That’s what the numbers say if we interpret them the way Mr. Son would have it. But no, that’s not what happened at all. It turns out that Speedtest – and similar test programs – can only measure the load they’re able to put on a network. When I run Speedtest on my phone, that limit is about ten Mbps, but when I run Speedtest on my Mac and only use the phone as hotspot, the limit goes up to 21 Mbps. LTE is work for the phone, and the Speedtest program is even more work, and there’s only so much the poor phone can do. In the trade we describe such numbers as “device limited” rather than “network limited.”

I can’t run OpenSignal in a hot-spot configuration using my Mac because it’s a mobile-only app, but I think it’s reasonable to surmise that there’s a fair bit of device effect in its figures. Running it from an old iPad in hotspot mode I see the download go up to 13 Mbps, and this iPad shows Speedtest figures on my real Wi-Fi router less than half the Mac’s speed.

Perhaps the point I’m trying to make here will be more clear if you conduct a simple experiment on your own. Try running Speedtest on the devices you have over the Wi-Fi network you have in your own home. The fair way to do it is to sit in the same place and run tests from the Speedtest mobile phone app (available from the app store of your choice), a tablet, and a laptop. When I do this, I get the following numbers:

Device                       Download       Upload
———————————————-
Phone                         21.38              8.46
iPad                            27.58              11.28
Laptop                        57.32              11.25

What’s the speed of my network connection?  As it turns out, my laptop is a top of the line model that runs much faster than my Internet connection so the figures it gets are close to 100% of network capacity; they’re higher than the carrier’s advertised speed claim, which is 50 down and 10 up. The other numbers are meaningful in terms of my experience on these other devices, but only a fool would consider them accurate measurements of network capacity.

This should show you that the measurements you can get from consumer-grade network testing apps say more about your devices than about the network itself. When these figures are reported as average and peak, peak is the better measurement, but unless you average the peaks throughout the day, you aren’t seeing the effects of network congestion. Akamai reports an “Average Peak” speed that does this, and it’s the most accurate measure of network capacity you can get from public test data.

The effect of device speeds on network performance measurements is important because the US was the first significant mover toward LTE: Verizon launched LTE service in December 2010 and AT&T followed about a year later; my Skyrocket phone is the first LTE device AT&T offered. The nations that rank higher than the US on the OpenSignal chart, such as Japan, have much less history. Softbank didn’t launch LTE service until 2012, and it deployed very slowly. So not only was it able to sell faster devices, it was able to deploy cheaper base-stations at higher density.

But more importantly, there is every reason to believe that there are more devices like the Samsung Galaxy S2 in America’s LTE test data than there are  Galaxy S5, the newest and fastest model, relative to the device population in countries like Japan that lagged the US in adopting LTE. The first mover is disadvantaged in device-dependent, crowd-sourced test data, and that’s who the US is. Europe was even slower to adopt LTE, so it’s no surprise that OpenSignal would rank Sweden higher than the US even if the networks in the two countries were exactly the same.

So even if we buy OpenSignal’s crowd-sourced measurement system as a valid yardstick, we should realize that comparing America’s LTE network speeds three years after initial rollout to other networks one or two years after theirs is fundamentally dishonest.  America’s larger broadband service providers are now preparing for their second generation LTE Advanced rollout, which will increase speed by making more spectrum and more towers available and also by riding on faster devices.

Even the OpenSignal data says that America’s LTE networks are more widely deployed than Japan’s: Verizon’s LTE users spend 76.8% of their connection time on an LTE network, while Softbank users in Japan only see LTE 71.1% of the time.  Mr. Son didn’t mention that.

The bottom line is that it’s unfair to use performance figures to punish first-movers.  The fastest computer is always going to be the newest one, and the same goes for networks. The countries with the fastest LTE networks are those who waited the longest to deploy LTE, and the networks with the fastest LTE speeds are those with the smallest number of users and the fastest handsets. In the early days of Verizon’s and AT&T’s LTE networks, 20 Mbps download speeds were common; after the next round of upgrades, speeds will go up to the next plateau, probably 40 – 80 Mbps. This is the way technology works and it’s the reason no serious person will ever try to pass crowd-sourced test data off as an accurate reflection of network capacity.