Leveraging Prometheus and Grafana to monitor a distributed application

Now that we have learned how to instrument an application service to expose Prometheus metrics, it’s time to show how we can collect the metrics and forward them to a Prometheus server where all metrics will be aggregated and stored. We can then either use the (simple) web UI of Prometheus or a more sophisticated solution such as Grafana to display important metrics on a dashboard.

Unlike most other tools that are used to collect metrics from application services and infrastructure components, the Prometheus server takes the load of work and periodically scrapes all the defined targets. This way, applications and services don’t need to worry about forwarding data. You can also describe this as pulling metrics, versus pushing them.

This makes Prometheus servers an excellent fit for our case. We will now discuss how to deploy Prometheus to Kubernetes, followed by our two sample application services...