So someone finally said it. A very notable someone at that. In a recent Made by Google podcast, Monika Gupta, Senior Director of Product Management for Google Silicon Teams, said what a number of people in tech already knew but had avoided saying because it was totally against popular perception:

Specs and benchmarks are handy, but it is functionality that counts.

Responding to criticism of Google’s Tensor chip for its poor benchmarking scores, Gupta was fairly direct:

I think classical Benchmarks served a purpose at some moment in time, but I think the industry has evolved since then… classical benchmarks were authored at a time when AI and phones didn’t even exist. They may tell some story, but we don’t feel like they tell the complete story… What we benchmark are the actual software workloads that we are running on our chip. Then we strive with every generation of the Tensor chip to make them better, whether it’s better quality, better performance, or lower power.

Want to know how good a phone is? Check them benchmarks

Some might find traces of irony in that statement, for benchmarks and tech specs really came into the smartphone review and analysis picture with the arrival of Android. Before that, in the age of Nokia and BlackBerry, it was rare to know the name and speed of the processor and RAM in a device, let alone Benchmark Scores of processors. The only specs that were considered generally were camera megapixels, display size (resolution was not as much of a factor), and to some extent, battery size.

Compare that to today, when just about everything inside a phone has some sort of benchmark test – the processor, the display, the cameras, the battery…you name it, and you will find it. These existed in the past as well, but in recent times, they have actually become the star performers in the smartphone show instead of being side actors. Today, it is common to see a brand start sharing benchmark scores of a phone’s processor, DXO ratings of its camera, and/or details of its display rating well before its release.

One can understand where the brands are coming from in this regard. These benchmarks and ratings are seen as “third party” and objective proof of performance and quality and often are a way of showing how a product is the best in a certain category or better than its rivals. And that certainly is a fair enough approach. After all, benchmarks and ratings are devised to test components, so how well they do in that department cannot be ignored.

Benchmark blues: “They may tell some story, but not the complete story”

The problem comes when benchmarks are seen as the be-all and end-all as far as a device is concerned. A Formula One driver had once said that the numbers on a car were no good if it was not easy to handle and drive, and the same applies to smartphone benchmarks. As Gupta remarked: “They may tell some story, but we don’t feel like they tell the complete story.” A processor might run up great benchmark scores, but it cannot compensate for buggy software. A high DXO rating for a camera might disguise sluggish processing. There are only so many benchmarks and ratings that can do. They are like test conditions, not real-life ones. What’s worse, brands have now started focusing on great benchmark scores instead of genuine consumer experience. Indeed, the benchmark score has become an important part of any brand launch presentation!

To its credit, Google has been trying to break out of this benchmark-driven system for a while. When it took over Motorola, it came out with devices like the Moto G and the Moto X, which stressed smooth and innovative performance rather than specs. Even when it seemed to have been sucked into the spec wars with the Pixel range, Google came out with ‘A’ variants of the Pixel that delivered good performance with relatively modest spec sheets. And when it moved to its own Tensor processor platform for the Pixels last year, the search giant made it clear that it was prioritizing smart functionality over ‘power.’ Power that was defined by benchmark scores that is.

Tensor: stuff to make benchmarks tense

The Pixels have been taking the flak for not being as powerful as devices driven by flagship chips from Qualcomm and MediaTek. However, a point to note here is that they tend to lag behind only in very specific conditions – say, at maxed-out settings of a high-end game or while processing a long video. In most mainstream usage tasks, the Pixels are as good as any Android flagship, if not better, and come with a number of smart features that many flagships lack. Yes, they might not load games as swiftly as some other flagships and might drop the odd frame here and there, but the difference is not as calamitous as benchmark scores indicate. As an executive from a tech brand once pointed out, “Should benchmark scores matter if the phone is working smoothly for a user? It is kind of unfair to call a phone underpowered based on the name of its processor or its benchmark scores. It should be about how well it works, not what its benchmark scores are.”

This is not to say benchmarks are utterly useless and a waste of time. They are invaluable in providing us with a measure of performance and also a scale of comparison. However, they need to complement the consumer tech narrative rather than drive it. Benchmarks are a little like academic examinations – they do provide a measure of what a person knows but are by no means irrefutable proof of a person’s knowledge. Even Einstein flunked examinations. We have frankly lost count of the number of devices that boasted impressive benchmark scores but stuttered and stumbled in the real world.

Google has taken a bold step forward in a benchmark-obsessed world by opting for smartness rather than the sheer speed with its Tensor chips. It is wonderful to see it back it as well, even as some tech pundits rain scorn on its benchmark scores. “What we benchmark are the actual software workloads that we are running on our chip,,” Gupta said in the podcast. We think she has a point. At the end of the day, it should be about how well something works. Phones with high benchmarks and mediocre/buggy performance are akin to dishes that have excellent ingredients and presentation but still end up tasting odd. Benchmarks are undoubtedly useful, but it is time we started to look beyond them.