Samsung Isn’t The Only Manufacturer Cheating On Android Benchmarking Tests

It looks like Samsung isn’t the only Android hardware manufacturer gaming the benchmarking tests. A fascinating look by Anandtech shows that a large majority of handset manufacturers are playing the same type of dirty tricks. The site built a comprehensive chart (hysterically named “I Can't Believe I Have to Make This Table") that shows how a number of devices are rigged to cheat certain benchmarking tests:

We started piecing this data together back in July, and even had conversations with both silicon vendors and OEMs about getting it to stop. With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It's possible that older Motorola devices might've done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.

And the report rightfully questions what the manufacturers gain by playing the system:

The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 - 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can't stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors. Whether the OEMs choose to change or not however, we’ve seen how this story ends. We’re very much in the mid-1990s PC era in terms of mobile benchmarks. What follows next are application based tests and suites. Then comes the fun part of course. Intel, Qualcomm and Samsung are all involved in their own benchmarking efforts, many of which will come to light over the coming years. The problem will then quickly shift from gaming simple micro benchmarks to which “real world” tests are unfairly optimized which architectures.

The story is a bit technical, but is a fascinating read to see how much trouble goes into cheating on these tests, and the work that Anandtech did to find out exactly how it was done. Apple’s own Phil Schiller took to Twitter yesterday calling Samsung’s cheating on its new Galaxy Note 3 “shenanigans.” But it looks like the issue spreads to most of the Android hardware manufacturer ecosystem. At least Apple lets the 64-bit A7 chip do all the talking without playing tricks. For other industry news today, see: Apple's Eddy Cue Opens Up About The iTunes Festival And iTunes Radio, AppleCare Staff Begins OS X Mavericks Training, and Apple Is Sitting On 10 Percent Of All Corporate Cash.

Related articles