Summary

In this chapter, we saw the real value of data observability for data engineers, where they troubleshoot or even firefight issues in their day-to-day jobs. Days or weeks of tedious manual checks can be avoided by adding proactive and at-the-source observability. The observability metrics that are collected by applications moving, reading, and transforming the data are great assets for performing analyses in case any issues occur.

Furthermore, we have seen that the more observable the system is, the easier it is to evaluate the impact of any issue, allowing the team to work efficiently on what requires the most attention. The in-context collected metrics allow us to easily overview the content of the data through the lineage so that we can correctly identify the faulty application or data and fix it faster.

This is only one of the main advantages of implementing data observability. In the next chapter, we will explore how data observability can be used to optimize pipelines...