The rise of big data as a corporate asset

You don’t need to look too far or too hard to hear about the many ways that big data and data analytics are transforming organizations and having an impact on society as a whole. We hear about how companies such as TikTok analyze large quantities of data to make personalized recommendations about which video clip to show a user next. We also read countless articles about how the new generation of chatbots (like ChatGPT from OpenAI or Bard from Google) have been trained on massive datasets, and as a result, are able to have human-like conversations on a wide range of topics. We experience how companies like Amazon and Netflix are able to recommend products or videos we may be interested in, based on our purchase and viewing history. All of these companies have innovated and added customer value by performing complex analyses on very large datasets.

We also see the importance of data in large companies, as demonstrated by those companies creating a new executive C-level position – the Chief Data Officer (CDO). According to an article (https://hbr.org/2021/08/why-do-chief-data-officers-have-such-short-tenures) in the Harvard Business Review, the role of CDO was first established by Capital One (a technology-driven U.S. bank) in 2002. By 2012, it was estimated that 12% of firms had a CDO according to a NewVantage Partners survey, and by 2021, this had grown to 65% of firms having a CDO.

There is no doubt that data, when harnessed correctly and optimized for maximum analytic value, can be a game-changer for an organization. At the same time, those companies that are unable to effectively utilize their data assets risk losing a competitive advantage to others that do have a comprehensive data strategy and effective analytic and machine learning programs.

Organizations today tend to be in one of the following three states:

• They have an effective and modernized data analytics and machine learning program that differentiates them from their competitors.
• They are conducting proof of concept projects to evaluate how modernizing their analytic and machine learning programs can help them achieve a competitive advantage.
• Their leaders are having sleepless nights worrying about how their competitors are using new analytics and machine learning programs to achieve a competitive advantage over them.

No matter where an organization is in its data journey, if it has been in existence for a while, it has likely faced a number of common data-related challenges. Let’s look at how organizations have typically handled the challenge of ever-growing datasets.