Data Observability Techniques
As we have already learned, sometimes, data quality is still not considered a determining factor during the life cycle of data, or the design and development of new data pipelines.
When a data team has to design and build a new data pipeline, there are many aspects to focus on:
- Data sources – input and output
- Scalability, reliability, and performance
- Total Cost of Ownership (TCO)
- Security
- Operation and maintainability
- Compliance with data regulations
But, often, what is missing is the adoption of data quality and observability by design – in other words, defining the specifications regarding what and how to monitor from the beginning of the design of the new data pipeline. These expectations are often not well defined and agreed upon between producers and consumers and even though this might seem unusual, data teams often only develop this sense of need for data quality and observability over time –...
