A new report says US mobile networks are mediocre compared to lightning fast networks in France and the UK, unintentionally showing how hard it is to compare networks.  The report doesn’t rank networks simply by download speed, but by an index composed of speed, latency, error rate, and jitter (they call it “latency consistency” but it means variations in delay.) The data is limited to 27 networks in the US, Canada, UK, France, Italy, Germany, and Spain. Here’s the raw ranking, which I’ll explain the text following.

MobiNEX Mobile Network Ranking Q4 2015

Is This a Valid Study?

Since I wrote a research report on network quality in the G7 [note: that’s not a strikethrough, it’s a buggy link] a year and a half ago, I have fairly current data to compare to the report. I devised an index for comparing nations on broadband quality in my report, but it’s more inclusive since it includes both wired and mobile networks so I won’t mention it further.

Before I dive in, there’s one detail in the presentation of the report that’s defies reason. The authors found high error rates in US networks, and found this “a surprise given that many applications have been developed by U.S.-based start-ups and established companies, and the vast majority are hosted in the datacenters of U.S. Internet giants Amazon and Google.”

But error rates on wireless networks have almost nothing to do with where applications are developed and where services are hosted. The overwhelming majority of lost or damaged packets are impaired on the radio link between the handset and the radio tower. Any errors that take place on the wired network are typically too low to count. So this assertion raises a red flag about the technical acumen of the analysts.

Figure 59: G7 Mobile Speed Test Scores. Source: Akamai, Ookla, OpenSignal.

Another troubling methodological detail pertains to the source of the raw data, Open Signal. This is a crowdsourced data collection system that relies on apps installed on smartphones that periodically measure the network by performing crude tests. When I looked at Open Signal’s data in the G7 report in comparison with the predictions made by Ookla and Akamai,  found that Open Signal was the median for Italy, Canada, France, Germany, Japan, and the UK,  but significantly below the median for the US. As the chart shows, Akamai tends to show high speeds, Ookla shows low speeds, and Open Signal shows median speeds except in the US, where it shows low speeds.

This quirk suggests that the user population that has chosen to install Open Signal on its phones is not representative of the overall user population in the US for some reason that seems to be country-specific. But whatever the reason, it’s safe to assume that a comparison between the US and other countries based on Open Signal will give the US an artificially low ranking.

Why do Nations with Advanced Networks Have Low Speeds?

Open Signal’s ranking of the US and Japan – the two G7 nations with the most advanced mobile networks – placed them at the bottom of the G7, an unlikely event two years ago, when the rest of the world didn’t have LTE. Open Signal is primarily useful as a measurement of technology deployment, as it tells us what percentage of connections take place over LTE networks vs. non-LTE ones.

All other things being equal, we would expect LTE networks to outperform 3G networks, as the Akamai data show for Japan and the Ookla data show across the board except for Japan, weirdly. (One consistent finding in all three data sets is quite high speeds in France and the UK, which seems to be a related to the low speeds of their wired networks and their regulatory approaches. When wired broadband is condemned to the slow lane by regulation, the more competitive wireless networks up their game. But that’s another story.)

It’s possible to cross-correlate LTE coverage and speed with Open Signal, and when we do that we see that speeds tends to fall as coverage increases.

Figure 62: G7 OpenSignal LTE Speed as Function of Coverage Source: OpenSignal

The reason for this is very simple: Users will have the best network experience when they’re early adopters of smartphones of the most recent generation. When networks roll out technology upgrades, they only directly benefit users whose smartphones are ready to use the new tech. When the number of new phones is small, users will get a lightning-fast experience because they’re sharing a high capacity network with a small number of other users so there’s little contention for resources. The best signal of resource contention is, you guessed it, error rate. So that’s the answer to the error rate quandary: US users (and you’ll see the same effect in other high tech nations such as Japan and South Korea) are sharing leading edge networks with each other more heavily than users in other countries.

US Consumers are Heavy Users of Mobile Data

This heavy sharing is not simply a function of leading the pack to LTE and using the latest iPhones, it also shows up in the amount of data we download.

Figure 75: G7 estimated mobile data usage per household. Sources: Cisco, World Bank, and Nationmaster.

Except for Japan, we’re the heaviest users of mobile data in the G7: We use more than the UK, twice as much as Italy and four times as much as France. So this compounds the data speed problem. Not only are we sharing with more people, we’re sharing with people who do much more than Europeans do. Anecdotally, it appears than many continentals moved to smart phones mainly for text messaging, but that seems to be changing as they grow more comfortable with apps.

A Matter of Timing

This is a good time to be a smartphone user in Europe. Your networks are shiny and lightly-used, and you pretty much have LTE all to yourself if you have a newish phone. You had to wait two years longer than your American cousins for your network, but at long last you’re having the time of your life. In the US, our networks are contending well enough with our heavy appetites for data and we have the kind of LTE coverage that Europeans would love to have to the extent that they’ve embraced the new technologies.

An awful lot is taking place behind the scenes as all of us prepare for the next technology evolution toward 5G, especially field trials and testing. But that sort of thing doesn’t show up in crowdsourced testing right now. But be patient, it will. For the time being, the 10 megabits/second of download speed you have in the US isn’t getting in your way because that’s as fast as most web sites can serve up pages, even on wired networks. So ho hum.

UPDATE: A reader points out that one of the study’s authors, STL Partners, has ties to the industry: “STL is a London-based firm whose clients include Sprint and Deutsche Telekom, which owns T-Mobile and holds substantial shares in several European operators.” Oops.