Three principles of data observability

By definition, observability comes from the system itself and reporting its state to the outside world. In data observability, the root of the system is the data process. A process is almost always an application, so a script, a notebook, or a program, for instance. The application reports its inner activity, which allows the external observer to understand what happens inside the process. More than creating a data product output, a layer of data observability allows the application to produce data on its proper execution.

Data observability comes with three main principles:

Data observability = Contextual + Synchronous + Continuous validation

This is all about avoiding using unnecessary observations, avoiding data error cascade, and ensuring non-propagation of known issues.

Let’s explore these principles in detail:

• Observability must be put into context: This point is about contextualizing your indicators. Indeed, a...