Validating data outputs
When we have created a data pipeline, we need to validate the records produced. This validation involves checking whether our dataset is what we are expecting, and, if there are any outliers, we can reprocess them after identification. This process needs to be as automated as possible, so actions are implemented quickly without general manual checks.
Automating the result monitoring actions
When you have built your data pipeline and added the testing steps in the Workflow tests and messages section, you want to minimize the number of interventions you need to make. However, any action you have to take manually is a possible point of failure, so we need to focus on automating a call to action in situations where our checks fail.
The most effective method is to use Workflow Events from the Workflow Configuration menu. We can see the Configuration menu in the following screenshot:
Figure 9.5 – Workflow Events menu
In the...
