Setting up Grafana for monitoring
Grafana is an open source tool built to create visualizations and monitor data from other systems and applications. Together with Prometheus, it is one of the most popular DevOps tools due to its flexibility and rich features.
In this exercise, we will configure a Docker image to run Grafana and connect it to Prometheus. This configuration will not only give us the ability to explore the Airflow metrics even further but also the opportunity to learn in practice how to work with a set of the most popular tools for monitoring and observability.
Getting ready
Refer to the Technical requirements section for this recipe since we will handle it with the same technology.
In this recipe, I will use the same docker-compose.yaml file of Airflow and will keep the configurations from the Setting up StatsD for monitoring and Setting up Prometheus for storing metrics recipes, to connect them and proceed with the monitoring and observability architecture...
