Fundamentals of Data Observability

Perfect! You have set up new objectives for the marketing report data so Lucy, the marketing analyst we met in Chapter 1, is delighted. Your team is now aware of the expectations of the marketing team, and the trust relationship between the marketing and data engineering teams can be rebuilt. You are now thinking about ways your data engineering team will monitor quality, and it will start by analyzing the datasets you provide to the business in order to prove they are correct. Every day, you check the quality of the output table in a Snowflake database. Quickly, an error appears. The values in a column seem to have been divided by a factor of 10. What a shame! Lucy is already using the dataset. Hopefully, you can proactively contact her and avoid errors in her report. Nevertheless, you must now fix the issue. This won’t be a piece of cake… Looking at the central data catalog, you are already losing hope… How should you start...