Creating an insight dashboard for workflow monitoring
Creating a dashboard using the Insights tool gives you the ability to isolate any issues you need to be aware of and whether the performance of a workflow has changed. In this section, we will create an example dashboard that allows us to monitor the daily runtime performance of workflows. When trying to develop workflows following the DataOps principles, we need some method to monitor quality and performance, as described in the Confidence pillar from Chapter 3, DataOps and Its Benefits.
Creating a monitoring dashboard
The dashboard we will create has three charts:
- An area chart for the daily run count
- A line chart showing the average runtime of jobs
- A line chart showing the 90th percentile of the runtime for jobs
The completion success will split these three metrics to see whether the success of workflows causes any variation. We see the final dashboard in the following screenshot:
