Implementing distributed tracing

So far, you’ve created a solution with two microservices, the football trading microservice and the client microservice. Among other features, the trading microservice provides the ranking of players. The client microservice enhances the list of players by adding the ranking that was obtained from the trading microservice.

Distributed tracing emerges as a crucial tool as it offers a systematic approach to monitoring, analyzing, and optimizing the flow of requests between microservices. Distributed tracing is a method of monitoring and visualizing the flow of requests as they propagate through various components of a distributed system, providing insights into performance, latency, and dependencies between services.

In this recipe, you will learn how to enable distributed tracing for your microservices, export the data to Zipkin, and access the results.

Zipkin is an open source distributed tracing system that helps developers trace, monitor, and visualize the paths of requests as they travel through various microservices in a distributed system, providing valuable insights into performance and dependencies. What you will learn about Zipkin in this recipe can be easily adapted to other tools.

Getting ready

In this recipe, we’ll visualize the traces using Zipkin. You can deploy it on your computer using Docker. For that, open your terminal and execute the following command:

docker run -d -p 9411:9411 openzipkin/zipkin

The preceding command will download an image with an OpenZipkin server, if you don’t have one already, and start the server.

We’ll reuse the trading service we created in the Using probes and creating a custom health check recipe. If you haven’t completed it yet, don’t worry – I’ve prepared a working version in this book’s GitHub repository at https://github.com/PacktPublishing/Spring-Boot-3.0-Cookbook/. It can be found in the chapter3/recipe3-4/start folder.

How to do it…

Let’s enable distributed tracing in the existing trading service and create the new client service. For the new client service, we’ll need to ensure that distributed tracing is enabled as well. Before starting, ensure that your OpenZipkin server is running, as explained in the Getting ready section:

1. Start by enabling distributed tracing in the trading microservice you created in the Using probes and creating a custom health check recipe:
   1. For that, open the pom.xml file and add the following dependencies:

   <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-tracing-bridge-otel</artifactId>
   </dependency>
   <dependency>
      <groupId>io.opentelemetry</groupId>
      <artifactId>opentelemetry-exporter-zipkin</artifactId>
   </dependency>

   2. The first dependency is a bridge between Micrometer and OpenTelemetry. The second dependency is an exporter from OpenTelemetry to Zipkin. I’ll explain this in more detail in the How it works… section.
   3. Now the application can send the traces to Zipkin. However, before running the application, you’ll need to make some adjustments. Open the application.yml file in the resources folder and add the following setting:

   management
       tracing:
           sampling:
               probability: 1.0

   4. By default, sampling is only set to 10%. This means that only 10% of traces are sent. With this change, you will send 100% of the traces.
   5. In the same application.yml file, add the following configuration:

   spring:
       application:
           name: trading-service

   This change is not mandatory but helps identify the service in distributed tracing.

2. Next, create the ranking endpoint in the football trading microservice that will be consumed by the client microservice. For that, in FootballController, create the following method:

   @GetMapping("ranking/{player}")
   public int getRanking(@PathVariable String player) {
       logger.info(«Preparing ranking for player {}», player);
       if (random.nextInt(100) > 97) {
           throw new RuntimeException("It's not possible to get the ranking for player " + player
                 + " at this moment. Please try again later.");
       }
       return random.nextInt(1000);
   }

   To simulate random errors, this method throws an exception when a random number from 0 to 99 is greater than 97 – that is, 2% of the time.

3. Next, create a new application that will act as the client application. As usual, you can create the template using the Spring Initializr tool:
   • Open https://start.spring.io and use the same parameters that you did in the Creating a RESTful API recipe of Chapter 1, except change the following options:
     • For Artifact, type fooballclient
     • For Dependencies, select Spring Web and Spring Boot Actuator
   • Add dependencies for OpenTelemetry and Zipkin, as you did for the football trading service application. So, open the pom.xml file and add the following dependencies:

     <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-tracing-bridge-otel</artifactId>
     </dependency>
     <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-zipkin</artifactId>
     </dependency>

4. In the client application, add a RESTful controller:
   1. Name it PlayersController:

   @RestController
   @RequestMapping("/players")
   public class PlayersController {
   }

   2. This application must call the trading service. For that, it will use RestTemplate. To achieve the correlation between service calls, you should use RestTemplateBuilder to create RestTemplate. Then, inject RestTemplateBuilder into the controller’s constructor:

   private RestTemplate restTemplate;
   public PlayersController(RestTemplateBuilder restTemplateBuilder) {
      this.restTemplate = restTemplateBuilder.build();
   }

   3. Now, you can create the controller method that calls the trading service of the other application:

   @GetMapping
   public List<PlayerRanking> getPlayers() {
      String url = "http://localhost:8080/football/ranking";
      List<String> players = List.of("Aitana Bonmatí", "Alexia Putellas", "Andrea Falcón");
      return players.stream().map(player -> {
         int ranking = this.restTemplate.getForObject(url + "/" + player, int.class);
         return new PlayerRanking(player, ranking);
      }).collect(Collectors.toList());
   }

5. Configure client application tracing in the application.yml file:

   management:
       tracing:
           sampling:
               probability: 1.0
   spring:
       application:
           name: football-client

   As you did in the trading service, you should set sampling to 1.0 so that 100% of the traces will be recorded. To distinguish the client application from the trading service application, set the spring.application.name property to football-client.

6. To avoid port conflicts with the trading application, configure the client application so that it uses port 8090. To do that, add the following parameter to the application.yml file:

   server:
     port: 8090

7. Now, you can test the application. Call the client application; it will make multiple calls to the trading service. To make continuous requests to the client application, you can execute the following command in your terminal:

   watch curl http://localhost:8090/players

8. Finally, open Zipkin to see the traces. For that, go to http://localhost:9411/ in your browser:

Figure 3.3: The Zipkin home page

On the home page, click RUN QUERY to see the traces that have been generated:

Figure 3.4: Root traces in Zipkin

On this page, you will see that the traces from the client application are root traces. Since we introduced a random error, you will see that there are failed and successful traces. If you click the SHOW button for any of these traces, you will see the traces of both RESTful APIs. There will be a main request for the client service and nested requests for the trading service:

Figure 3.5: Trace details, including nested traces

You can also view the dependencies between services by clicking on the Dependencies link on the top bar:

Figure 3.6: Viewing the dependencies between services in Zipkin

Here, you can see the dependencies between the football-client application and the trading-service application.

How it works…

Micrometer is a library that allows you to instrument your application without dependencies with specific vendors. This means that your code won’t change if you decide to use another tool, such as Wavefront, instead of Zipkin.

The io.micrometer:micrometer-tracing-bridge-otel dependency creates a bridge between Micrometer and OpenTelemetry, after which the io.opentelemetry: opentelemetry-exporter-zipkin dependency exports from OpenTelemetry to Zipkin. If you want to use another tool to monitor your traces, you just need to change these dependencies, without any additional code changes.

The default address to send traces to Zipkin is http://localhost:9411. That’s why we didn’t need to configure it explicitly. In a production environment, you can use the management.zipkin.tracing.endpoint property.

In this recipe, we used RestTemplateBuilder. This is important as it configures RestTemplate by adding the tracing headers to the outgoing requests. Then, the target service gathers the tracing headers that can be used to nest the traces in the called application to the root trace from the client application. In reactive applications, you should use WebClient.Builder instead of RestTemplateBuilder.

In this recipe, we configured 100% sampling. This means that we send all traces to the tracing server. We did this for learning purposes; normally, you shouldn’t do this in production as you can overload the tracing server by, for example, deploying a server via Zipkin or ingesting a lot of data if you’re using a managed service in the cloud. The amount of data that’s ingested directly affects monitoring systems – that is, the more data you ingest, the more it will cost you. However, even if you deploy your own tracing server, you will need to scale up as well. So, either way, it can increase your overall cost. In a large-scale system, having a sampling rate of 10% is more than enough to detect issues between services as well as understand the dependencies between the components.

There’s more…

Micrometer tracing creates spans – that is, units of work or segments of a distributed trace that represent the execution of a specific operation, for each request. Spans capture information about the duration, context, and any associated metadata related to the respective operation.

You can create a span by starting an observation using the ObservationRegistry component. For instance, say TradingService has different important parts that you want to trace, such as Collect data and Process data. You can create different spans for those in your code.

To implement this, you will need to inject ObservationRegistry into your controller using the Spring Boot dependency container. For that, you need to define the ObservationRegistry parameter in the controller’s constructor:

private final ObservationRegistry observationRegistry;
public FootballController(ObservationRegistry observationRegistry) {
        this.observationRegistry = observationRegistry;
}

Then, you must create the observations in the code:

@GetMapping("ranking/{player}")
public int getRanking(@PathVariable String player) {
   Observation collectObservation = Observation.createNotStarted("collect", observationRegistry);
   collectObservation.lowCardinalityKeyValue("player", player);
   collectObservation.observe(() -> {
      try {
          logger.info("Simulate a data collection for player {}", player);
          Thread.sleep(random.nextInt(1000));
      } catch (InterruptedException e) {
          e.printStackTrace();
      }
   });
   Observation processObservation = Observation.createNotStarted("process", observationRegistry);
   processObservation.lowCardinalityKeyValue("player", player);
   processObservation.observe(() -> {
            try {
                logger.info("Simulate a data processing for player {}", player);
                Thread.sleep(random.nextInt(1000));
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        });
        return random.nextInt(1000);
    }

Note that the observations include the player with lowCardinalityKeyValue to facilitate finding spans through this data.

Note

Some parts of the code have been removed for brevity. You can find the full version in this book’s GitHub repository at https://github.com/PacktPublishing/Spring-Boot-3.0-Cookbook/.

Now, in Zipkin, you can see the custom spans nested in trading-service:

Figure 3.7: Custom spans in Zipkin

The trading-service span contains two nested spans, and both have a custom tag that specifies the player’s name.