Defining Rules on Indicators

In the previous chapters, we saw how you could collect events synchronously in your data applications. We also discussed what contextual information you need in order to draw the big picture of what’s happening inside the applications.

Now that you have a lot of contextual information, it is high time to turn it into actionable insights. The metrics you collect during the pipeline execution need to reassure all the stakeholders about the proper execution of the data applications. All the observers of the pipeline need to be informed about how the data pipeline is behaving.

To maintain the trust of data producers and data consumers, we will introduce the concept of expectations, which will define what the engineer needs to achieve in order to keep the pipeline in good shape. These expectations, composed of metrics and rules, will act as sensors to know whether the applications are working as expected or not.

These rules are a key component...