Computing observability metrics
The following data observability elements are known as data quality metrics. In this category, we will group everything we consider to be observability metrics. These observations are statistics related to the data you manipulate:
- Distribution observations: Minimum, maximum, mean, standard deviation, skewness and kurtosis, quantiles, and so on
- Categorical stats: Number of categories, percentage of each category, and so on
- Completeness observations: Number of rows and number of missing values
- Freshness information: Timestamp of the data itself
- KPIs: Key performance indicators and other custom metrics worth checking, for technical or business purposes
The metrics you compute depend on the circumstances and need to be linked to the context where they were computed. Those metrics can change following the usage of the data, the filters you applied, and the application run. Figure 4.7 shows an example of multiple contexts for...
