I love data. I have dedicated my life to the analysis and communication of it. But our society has a data problem. We’ve gone overboard. In some cases, we may love data too much.

In nearly every part of society, the obsession with collecting data and assessing people and institutions based on those measurements has gone too far. From schools to board rooms, people in power increasingly use dubious metrics to make decisions, when they would be better off using the powers of logic, intuition, and expertise.

In his new book The Tyranny of Metrics, historian Jerry Muller explains how we got to this point, and highlights some of the most egregious examples of our over-reliance on data. He also offers guidelines for when measurement makes sense, and when it risks leading us astray.

Muller’s story begins in Victorian England. In 1862, a period when public-sector reform was all the rage, parliament changed the law so that government schools would be funded based on performance. The government would collect data on how well students performed on tests of reading, writing, and arithmetic, and take away funding from schools that performed poorly. The idea was to establish a bottom line for schools that would force them to operate more like businesses. (Similar arguments are made today.)

There were several problems with this. As cultural critic and school inspector Matthew Arnold pointed out at the time, performance on tests was mostly determined by students’ family backgrounds, not the quality of teaching at the school. Also, rather than encouraging curiosity or social skills, teachers were incentivized to teach to the test.

The collection of data to inform “pay for performance” in English schools was a portent of things to come. From the 1860s through today, the use data to assess and incentivize spread like wildfire across industries, spurred by the rise of “scientific management“—the theory of how to get employees to work most efficiently. Management gurus frequently espouse the idea that you can’t manage what you don’t measure.

When used carefully, data collection is extremely valuable. Data have been particularly useful for identifying employees who commit fraud and help the police identify crime hotspots. Yet the obsession with data often leads people to false conclusions. Modern management theory emphasizes that the measurement and monitoring of data are keys to improvement. As a result, in many countries, the police are now assessed by their ability to lower Crime Rates, doctors by how long they can keep people with certain conditions alive, and CEOs by their ability to boost their company’s stock price.

At first glance, these measures may seem reasonable. But Muller shows that if we dig deeper, the data have unintended consequences that can make people perform worse. A focus on crime rates incentivizes the police to misreport the severity of crimes to improve their statistics. Assessing doctors on their ability to keep patients alive leads them to take on fewer risky patients. Bonuses based on share price make CEOs focus on short-term actions that may be harmful in the long run.

These examples are only the tip of the iceberg. A large body of research (pdf) supports Muller’s argument that data-based incentives often backfire. Not only do they cause people to focus too much on what is measured, but they can also strip people of their intrinsic motivation to do a good job.

What’s the alternative? Many people might say you just need better numbers. Muller thinks that this just digs a deeper hole. At some point, the collection of increasingly complex data could leave no time for any other work. He believes that in many situations the better alternative is old-fashioned judgement. Rather than assess an employee by some complex equation, managers should use their knowledge of context to make a more holistic evaluation. This is easier when the manager has some experience with the job they are assessing. For this reason, Muller recommends companies and institutions promote from within.

The ultimate point of Muller’s books is that “measurement is not an alternative to judgment: measurement demands judgement.” The decision to collect data is often expensive and almost always fraught with the possibilities of misuse. More data isn’t always better. More thoughtful analysis always is.

Source: Qz