Data observability in IT observability

Observability is not a new asset in the monitoring toolkit. Indeed, making a system observable is already implemented by the operational and DevOps community to report on application or system availability. Observability helps us to monitor complex distributed systems by collecting logs, metrics, or traces.

If we take observability in the broadest sense, every component of the data pipeline can be monitored — not only the data, but the application, the system, or even the users. Data observability helps us to build the bridge between the data engineering world and application and system observability, with some common elements allowing a deeper analysis of data issues in the context of full observability.

Besides the problems that come from the data itself, other issues may arise — for instance, from an unresized cluster, a failing or outdated application, and so on. The combination of all kinds of observability can increase...