Data Quality and Observability
Have you ever wasted time or money because you made a decision based on incorrect data? Or because someone had a different definition of a key performance indicator (KPI)? If you have, you are familiar with the importance of data quality. In this chapter, we will cover not only the problem of data quality but also some solutions in the form of tools and techniques that can help you overcome this problem.
You will learn about three main areas of data quality: the potential data quality issues in source systems, the issues in infrastructure and pipelines when moving data between systems, and the issues related to data governance. You will also learn why data lineage and understanding the journey of your data as it moves through systems is a crucial part of guaranteeing data quality.
After learning about the problem of data quality, we will investigate data observability as a solution to addressing data quality issues. We will then discuss how data...
