the Future of data science is software engineering

The role of data scientist will change in the next 5 years. The bulk of data science projects entail the same subjects: forecasting and solutions being enriched with image or voice recognition. These projects are available out-of-the-box through prebuilt AI such as the Cognitive Services from Microsoft or Google’s AI Building Blocks, with many more appearing on other clouds and marketplaces. The next step from prebuilt AI is usually custom code projects written in Python or R. Teams build their own machine learning models but even here technologies such as AutoML have simplified the process.

Being a traditional data scientist myself, this is a hard pill to swallow. With data scientists being expensive and difficult to find (although this shortage is decreasing each year – source) there are only a handful of companies that have the budget to build a data science team… and with mixed results.

As discussed in my MLops article the ROI for data scientists is difficult to achieve. Main challenges are that models do not make it to production, cannot be standardized and (despite the promise of autonomous machines) require too much manual, expensive labour. From a non-technical perspective most models are not applicable to the business: they prove hypotheses, but cannot be applied to the company’s product or solution. The models do not add value for the customers, or for the company internal.

Leave a Comment