After the advent of databases in the late 1970s and early 1980s, databases were treated as a central source of truth (SOT) and designed to record transactions and produce daily, weekly, and monthly financial reports. These are largely termed online transaction processing (OLTP) systems.

In the late 1980s, businesses felt the need to understand how their business was performing and investigate any changes to sales, production, revenue, or any other important aspects of the business so that they could run their businesses more efficiently. But in order to conduct this investigation, they had to run complex queries across all tables in their database and be able to slice and dice the data to dig deeper into it. They also had to aggregate values in order to find totals and averages across a period of time. A relational model that spread data across multiple tables was needed to aggregate and join data across these tables. As a result...

As enterprises get bigger, the ability to be agile and competitive becomes challenging. Some enterprises have large departments for sales, marketing, engineering, and so on. Many large corporations split their global businesses into regions or zones. Each of these sub-organizations operates as an independent unit. They have their own business complexity, analytics needs, and speed at which analytical output is required. They choose their tools according to their requirements. But at the end of the day, they are asked to move their data to a central lakehouse or data warehouse for enterprise-wide analytics, which uses a specific set of tools and mandates a certain data format. This strategy has multiple challenges:

• The sub-organizations are forced to use the tools in the central analytical system to perform their analytics.
• The sub-organizations then start building their own local analytical platforms to speed up their...

Historically, data has always been treated as a backend. It was used by the middle tier and then surfaced to the frontend. Applications did not do a lot with the data other than aggregating and presenting it with better visuals. Relational database systems also ensured that data adhered to a schema and format and that all mandatory fields were populated. As a result, applications received quality data and had to do minimal checks on quality. But with semi-structured data, this equation changes. Semi-structured data does not comply with fixed schemas and rules of how data is formatted and populated. Advanced analytics, ML, and big data analytics need a lot of processing on the data before it’s consumed by any algorithm and application. ML algorithms provide exponentially accurate output as the volume of quality data increases.

In a paper published in 2001 (https://homl.info/6), Microsoft researchers Michele Banko and Erik Brill showed that different ML algorithms performed...

One of the concepts defined in https://www.thoughtworks.com/what-we-do/data-and-ai/data-mesh is the concept of data domains. Data domains are defined as a logical grouping of data and teams aligned with certain business domains, such as sales, marketing, or production. While each of these domains may have multiple data products, all the teams and the data used to build these products fall under the same domain. This domain team is responsible for managing and maintaining the data in their domain. This is described as domain ownership.

However, in reality, we have found that adopting a domain as a concept could be challenging for many companies as every company has its own structure. For example, large global companies that run their business in different geological zones have sales, marketing, production, and local finance departments in every location. Each of these departments works independently based on their local market and country requirements. Clubbing all sales...

As we learned in the Discovering the challenges of modern-day enterprises section, having a central data lake or data warehouse has several disadvantages, especially for large organizations. In the previous section, we learned about changes in data processing requirements driven by ML and advanced analytics and how data now needs to be treated like a product with its own complete life cycle.

To explain a data mesh in one sentence, a data mesh is a centrally managed network of decentralized data products. The data mesh breaks the central data lake into decentralized islands of data that are owned by the teams that generate the data. The data mesh architecture proposes that data be treated like a product, with each team producing its own data/output using its own choice of tools arranged in an architecture that works for them. This team completely owns the data/output they produce and exposes it for others to consume in a way they deem fit for their data. Here...

We saw in this chapter how data analytics evolved over time as technology advanced and as business needs changed. One of the main objectives of walking through this history is to realize that, once again, we are at the cusp of a change. Data-driven organizations are putting pressure on data products to deliver faster innovation to keep the company ahead of the competitive curve. We also saw how data preprocessing has become critical to modern-day analytics, which uses machine learning for accurate predictions and forecasting. Clean, curated data itself becomes like a product that other products can consume to get innovative insights. This drives the need for a more collaborative and agile analytical environment where data can be discovered and used to build data products, as opposed to the centralized dashboards and reports of the past. A data mesh is one of the ways to bring this agile and collaborative framework to life.

However, a data mesh is a long-term strategy and...