Ten years later, I still remember reading Tom Davenport’s argument that data scientists had the hottest job on the planet. It’s tough to forget an article with a juicy title like Data Scientist: The Sexiest Job of the 21^st Century. I wasn’t the only person it made an impression on. Since its 2012 publication in the Harvard Business Review, the research study has racked up more than 1,600 citations in other scholarly journals. Davenport is the prolific author of 17 books, co-founder of the International Institute for Analytics, a fellow at MIT and an originator of the term “attention economy.” He co-authored that 2012 HBR study with DJ Patil, who was later appointed by former U.S. president Obama as the White House’s first-ever chief data scientist. To mark the tenth anniversary of their original paper, the pair recently penned a sequel for HBR titled (what else?) Is Data Scientist Still the Sexiest Job of the 21^st Century? I interviewed Davenport to get his take on how the role of data scientist has evolved over the past decade, how effectively businesses use data today vs. in 2012, and how far we’ve come in addressing the data skills shortage.

Looking back

When Davenport and Patil wrote that landmark 2012 research paper, the word “sexiest” did not appear anywhere in their original title. “I think it was an editor’s idea, although I think we did have to sign off on it,” Davenport tells me. Davenport and Patil’s 2012 article did describe the occupation of data scientist as sexy, however, in terms of “having rare qualities that are much in demand … difficult and expensive to hire and … difficult to retain.” They defined data scientists as professionals who “bring structure to large quantities of formless data and make analysis possible,” then “communicate what they’ve learned and suggest its implications for new business directions.” That same think piece raised early warning flags about challenges that were already percolating in the nascent world of enterprise data science, including:

• data overload: “Companies are now wrestling with information that comes in varieties and volumes never encountered before”
• skills and talent shortage: “… demand has raced ahead of supply”
• soaring pay: “… salaries will be bid upward”
• no standardization of skills and education: “There are no university programs offering degrees in data science”

Another observation they made was that businesses were grappling with how to best utilize data scientists to their advantage. “There is little consensus on where the role (of data scientist) fits in an organization, how data scientists can add the most value, and how their performance should be measured,” Davenport and Patil wrote a decade ago. They also predicted that instead of coding skills, tomorrow’s data scientists would need to acquire soft skills to “communicate in language that all their stakeholders understand” and forge “close relationships with the rest of the business.”

What’s changed

In the 2022 sequel to their 2012 article, Davenport and Patil acknowledge some things haven’t changed at all. The high demand for data expertise. The tight supply of skills. The soaring salary levels. And according to new Gartner research, extracting business value from data remains a thorny issue today. Per Gartner, barely half of AI pilot projects at enterprises make it to production these days, and “organizations still struggle to connect the algorithms they are building to a business proposition.” In their new HBR piece, Davenport and Patil assert that a decade on from their original study, “many organizations don’t have data-driven cultures and don’t take advantage of the insights provided by data scientists.” But why? “Cultural change of any type always takes a long time,” Davenport tells me. “It’s happening, but it’s a lot slower than some people might have predicted.” One thing that’s vastly different from 2012, he says, is the current focus on data ethics, a phenomenon he believes was sparked by the Cambridge Analytica/Facebook scandal. “The social and political divisiveness (around) social media and the analytics of AI that drive like-minded people toward each other and send them the kinds of news that only supports their point of view, I don’t think anybody really anticipated how likely that was to happen.” As the fallout continues, Davenport anticipates more government regulation of data use and AI, although he expects Europe to enact legislation long before the U.S. Another big shift he’s noticed over the past decade? Companies have stopped trying to find all the data science expertise they need in one elusive “unicorn” candidate. Instead, they’re looking for a wider range of skills from a wider breadth of fields. He says today’s enterprises are creating multiple data-related roles within their organizations, including “translators and people who are working closely with the business to understand their needs.” “The key to value is deploying (algorithms) within the business and that takes a whole broad range of skills,” he adds. In 2022, just as in 2012, there’s still no universally recognized system of education, skills and certification for data-based professions. Davenport feels that more standardization would help businesses hire—and internally cultivate—crucial data talent. “I think ultimately that would be good. The times and the world sort of change faster than the certification processes do, but I think it would help a lot of organizations to have more standard classifications.”

The future of data

What does Davenport expect to see in analytics 10 years from now? He predicts machine learning will become even more automated than it is today, freeing up data scientists to focus less on building ML models and more on tackling business problems. He says a type of “business/technology hybrid” worker is already emerging in enterprise organizations, a trend that could one day produce more data/business chameleons than data science unicorns. “These people understand the technology, they’ve been working with it for a long time, but they apply it to solving business problems,” he explains. Before I bid Davenport goodbye, he offers a parting thought on the future of my own profession in this age of GPT-3, the AI-based program whose writing output (a movie screenplay and an academic research paper) has already outpaced mine. “There are some journalism things that have already been somewhat automated but not feature stories of the kind that you write,” he grins. “So you’re probably safe for a while.” Images: metamorworks/iStock; gremlin/iStock

The post Is data Scientist still the hottest job in IT? appeared first on expertIP.