Exporting metrics

As already shown, exporting metrics from your own application is easy, at least in principle. All your application needs to do is offer an HTTP endpoint that returns arbitrary metrics that can then be saved in Prometheus. In practice, this gets more difficult, especially when you care about the status of the Go runtime (for example, CPU and memory usage, Goroutine count, and so on). For this reason, it is usually a good idea to use the Prometheus client library for Go, which takes care of collecting all possible Go runtime metrics.

As a matter of fact, Prometheus is itself written in Go and also uses its own client library to export metrics about the Go runtime (for example, the go_memstats_alloc_bytes or process_cpu_seconds_total metrics that you have worked with before).

Using the Prometheus client in your Go application

You can get the Prometheus client library using go get, as follows:

$ go get -u github.com/prometheus/client_golang 

In case your application uses a dependency management tool (such as Glide, which we introduced in the preceding chapter), you will also probably want to declare this new dependency in your glide.yaml file and add a stable release to your application's vendor/ directory. To do all this in one step, simply run glide get instead of go get within your application directory:

$ glide get github.com/prometheus/client_golang 
$ glide update 

For security reasons, we will expose our metrics API on a different TCP port than the event service's and booking service's REST APIs. Otherwise, it would be too easy to accidentally expose the metrics API to the outside world.

Let's start with the event service. Setting up the metrics APIs does not require much code, so we will do this directly in the main.go file. Add the following code to the main function before the rest.ServeAPI method is called:

import "net/http" 
import "github.com/prometheus/client_golang/prometheus/promhttp" 
// ... 
 
func main() { 
  // ... 
 
  go func() { 
    fmt.Println("Serving metrics API") 
 
    h := http.NewServeMux() 
    h.Handle("/metrics", promhttp.Handler()) 
 
    http.ListenAndServe(":9100", h) 
  }() 
 
  fmt.Println("Serving API") 
  // ... 
} 

Now, compile your application and run it. Try opening the address at http://localhost:9100/metrics in your browser, and you should see a large number of metrics being returned by the new endpoint:

Page shown at localhost:9100/metrics

Now, make the same adjustment to the booking service. Also, remember to add an EXPOSE 9100 statement to both service's Dockerfiles and to recreate any containers with an updated image and the -p 9100:9100 flag (or -p 9101:9100 to prevent port conflicts).

Configuring Prometheus scrape targets

Now that we have two services up and running that expose Prometheus metrics, we can configure Prometheus to scrape these services. For this, we can modify the prometheus.yml file that you created earlier. Add the following sections to the scrape_configs property:

global: 
  scrape_interval: 15s 
 
scrape_configs: 
  - job_name: prometheus 
    static_configs: 
      - targets: ["localhost:9090"] 
  - job_name: eventservice 
    static_configs: 
      - targets: ["events:9090"] 
  - job_name: bookingservice 
    static_configs: 
      - targets: ["bookings:9090"] 

After adding the new scraping targets, restart the Prometheus container by running docker container restart prometheus. After that, the two new scraping targets should show up in the Prometheus web UI:

Prometheus web UI targets

Now, for the best part—remember the Grafana dashboard that you have created a few sections earlier? Now that you have added two new services to be scraped by Prometheus, take another look at it:

Gafana

As you can see, Grafana and Prometheus pick up metrics from the new services instantly. This is because the process_cpu_seconds_total and go_memstats_alloc_bytes metrics that we have worked with until now are actually exported by all three of our services since they're all using the Prometheus Go client library. However, Prometheus adds an additional job label to each metrics that is scraped; this allows Prometheus and Grafana to distinguish the same metrics coming from different scraping targets and present them accordingly.

Exporting custom metrics

Of course, you can also use the Prometheus client library to export your own metrics. These do not need to be technical metrics that reflect some aspect of the Go runtime (such as CPU usage and memory allocation), but it could also be business metrics. One possible example could be the amount of booked tickets with different labels per event.

For example, within the todo.com/myevents/bookingservice/rest package, you could add a new file—let's call it metrics.go—that declares and registers a new Prometheus metrics:

package rest 
 
import "github.com/prometheus/client_golang/prometheus" 
 
var bookingCount = prometheus.NewCounterVec( 
  prometheus.CounterOpts{ 
    Name:      "bookings_count", 
    Namespace: "myevents", 
    Help:      "Amount of booked tickets", 
  }, 
  []string{"eventID", "eventName"}, 
) 
 
func init() { 
  prometheus.MustRegister(bookingCount) 
} 

The Prometheus client library tracks all created metric objects in a package, a global registry that is automatically initialized. By calling the prometheus.MustRegister function, you can add new metrics to this registry. All registered metrics will automatically be exposed when a Prometheus server scrapes the /metrics endpoint.

The NewCounterVec function used creates a collection of metrics that are all named myevents_bookings_count but are differentiated by two labels, eventID and eventName (in reality, these are functionally dependent and you wouldn't really need both; but having the event name as a label comes in handy when visualizing this metric in Grafana). When scraped, these metrics might look like this:

myevents_bookings_count{eventID="507...",eventName="Foo"} 251 
myevents_bookings_count{eventID="508...",eventName="Bar} 51 

The Prometheus client library knows different types of metrics. The Counter that we used in the preceding code is one of the simpler ones. In one of the previous sections, you saw how a complex histogram was represented as a number of different metrics. This is also possible with the Prometheus client library. Just to demonstrate, let's add another metric—this time, a histogram:

var seatsPerBooking = prometheus.NewHistogram( 
  prometheus.HistogramOpts{ 
    Name: "seats_per_booking", 
    Namespace: "myevents", 
    Help: "Amount of seats per booking", 
    Buckets: []float64{1,2,3,4} 
  } 
) 
 
func init() { 
  prometheus.MustRegister(bookingCount) 
  prometheus.MustRegister(seatsPerBooking) 
} 

When being scraped, this histogram will be exported as seven individual metrics: you will get five histogram buckets (Number of bookings with one seat or less up to Four seats or less and Infinitely many seats or less), and one metric for the sum of all seats and sum of all observations, respectively:

myevents_seats_per_booking_bucket{le="1"} 1 
myevents_seats_per_booking_bucket{le="2"} 8 
myevents_seats_per_booking_bucket{le="3"} 18 
myevents_seats_per_booking_bucket{le="4"} 20 
myevents_seats_per_booking_bucket{le="+Inf"} 22 
myevents_seats_per_booking_sum 72 
myevents_seats_per_booking_count 22 

Of course, we will need to tell the Prometheus library the values that should be exported for these metrics when scraped by the Prometheus server. Since both metrics (amount of bookings and amount of seats per booking) can only change when a new booking is made, we can add this code to the REST handler function that handles POST requests on the /events/{id}/bookings route.

In the booking_create.go file, add the following code somewhere after the original request has been processed (for example, after the EventBooked event is emitted on the event emitter):

h.eventEmitter.emit(&msg) 
 
bookingCount. 
  WithLabelValues(eventID, event.Name). 
  Add(float64(request.Seats)) 
seatsPerBooking. 
  Observe(float64(bookingRequest.Seats)) 
 
h.database.AddBookingForUser(
   // ... 

The first statement adds the amount of booked seats (request.Seats) to the counter metric. Since you defined one label named event in the CounterVec declaration, you will need to call the WithLabelValues method with the respective label values (if the metric declaration consisted of two labels, you would need to pass two parameters into WithLabelValues).

The second statement adds a new observation to the histogram. It will automatically find the correct bucket and increment it by one (for example if three seats are added with the same booking, the myevents_seats_per_booking_bucket{le="3"} metric will be increased by one).

Now, start your application and make sure that Prometheus is scraping it at regular intervals. Take the time and add a few example records to your application. Also, add a few event bookings at the booking service; ensure that you do not create them all at once. After that, you can use the myevents_bookings_count metric to create a new graph in your Grafana dashboard:

>

Gafana graph

By default, Prometheus will create one time series per scraped instance. This means that when you have multiple instances of the booking service, you will get multiple time series, each with a different job label:

myevents_bookings_count{eventName="Foo",job="bookingservice-0"} 1 
myevents_bookings_count{eventName="Foo",job="bookingservice-1"} 3 
myevents_bookings_count{eventName="Bar",job="bookingservice-0"} 2 
myevents_bookings_count{eventName="Bar",job="bookingservice-1"} 1 

When displaying a business metric (for example, the number of tickets sold), you may not actually care at which instance each particular booking was placed and prefer an aggregated time series over all instances, instead. For this, you can use the PromQL function sum() when building your dashboard:

sum(myevents_bookings_count) by (eventName)