Centralized Configuration

In this chapter, we will learn how to use the Spring Cloud Configuration server to centralize managing the configuration of our microservices. As already described in Chapter 1, Introduction to Microservices, an increasing number of microservices typically come with an increasing number of configuration files that need to be managed and updated.

With the Spring Cloud Configuration server, we can place the configuration files for all our microservices in a central configuration repository that will make it much easier to handle them. Our microservices will be updated to retrieve their configuration from the configuration server at startup.

The following topics will be covered in this chapter:

• Introduction to the Spring Cloud Configuration server
• Setting up a config server
• Configuring clients of a config server
• Structuring the configuration repository
• Trying out the Spring Cloud Configuration server

Technical requirements

For instructions on how to install tools used in this book and how to access the source code for this book, see:

• Chapter 21 for macOS
• Chapter 22 for Windows

The code examples in this chapter all come from the source code in $BOOK_HOME/Chapter12.

If you want to view the changes applied to the source code in this chapter, that is, see what it took to add a configuration server to the microservice landscape, you can compare it with the source code for Chapter 11, Securing Access to APIs. You can use your favorite diff tool and compare the two folders, $BOOK_HOME/Chapter11 and $BOOK_HOME/Chapter12.

Introduction to the Spring Cloud Configuration server

The Spring Cloud Configuration server (shortened to config server) will be added to the existing microservice landscape behind the edge server, in the same way as for the other microservices:

Figure 12.1: Adding a config server to the system landscape

When it comes to setting up a config server, there are a number of options to consider:

• Selecting a storage type for the configuration repository
• Deciding on the initial client connection, either to the config server or to the discovery server
• Securing the configuration, both against unauthorized access to the API and by avoiding storing sensitive information in plain text in the configuration repository

Let's go through each option one by one and also introduce the API exposed by the config server.

Selecting the storage type of the configuration repository

As already described in Chapter 8, Introduction to Spring Cloud, the config server supports the storing of configuration files in a number of different backends, for example:

• Git repository
• Local filesystem
• HashiCorp Vault
• JDBC database

For a full list of backends supported by the Spring Cloud Configuration Server project, see https://cloud.spring.io/spring-cloud-config/reference/html/#_environment_repository.

Other Spring projects have added extra backends for storing configuration, for example, the Spring Cloud AWS project, which has support for using either AWS Parameter Store or AWS Secrets Manager as backends. For details, see https://docs.awspring.io/spring-cloud-aws/docs/current/reference/html/index.html.

In this chapter, we will use a local filesystem. To use the local filesystem, the config server needs to be launched with the Spring profile, native, enabled. The location of the configuration repository is specified using the spring.cloud.config.server.native.searchLocations property.

Deciding on the initial client connection

By default, a client connects first to the config server to retrieve its configuration. Based on the configuration, it connects to the discovery server, Netflix Eureka in our case, to register itself. It is also possible to do this the other way around, that is, the client first connecting to the discovery server to find a config server instance and then connecting to the config server to get its configuration. There are pros and cons to both approaches.

In this chapter, the clients will first connect to the config server. With this approach, it will be possible to store the configuration of the discovery server in the config server.

To learn more about the other alternative, see https://docs.spring.io/spring-cloud-config/docs/3.0.2/reference/html/#discovery-first-bootstrap.

One concern with connecting to the config server first is that the config server can become a single point of failure. If the clients connect first to a discovery server, such as Netflix Eureka, there can be multiple config server instances registered so that a single point of failure can be avoided. When we learn about the service concept in Kubernetes later on in this book, starting with Chapter 15, Introduction to Kubernetes, we will see how we can avoid a single point of failure by running multiple containers, for example, config servers, behind each Kubernetes service.

Securing the configuration

Configuration information will, in general, be handled as sensitive information. This means that we need to secure the configuration information both in transit and at rest. From a runtime perspective, the config server does not need to be exposed to the outside through the edge server. During development, however, it is useful to be able to access the API of the config server to check the configuration. In production environments, it is recommended to lock down external access to the config server.

Securing the configuration in transit

When the configuration information is asked for by a microservice, or anyone using the API of the config server, it will be protected against eavesdropping by the edge server since it already uses HTTPS.

To ensure that the API user is a known client, we will use HTTP Basic authentication. We can set up HTTP Basic authentication by using Spring Security in the config server and specifying the environment variables, SPRING_SECURITY_USER_NAME and SPRING_SECURITY_USER_PASSWORD, with the permitted credentials.

Securing the configuration at rest

To avoid a situation where someone with access to the configuration repository can steal sensitive information, such as passwords, the config server supports the encryption of configuration information when stored on disk. The config server supports the use of both symmetric and asymmetric keys. Asymmetric keys are more secure but harder to manage.

In this chapter, we will use a symmetric key. The symmetric key is given to the config server at startup by specifying an environment variable, ENCRYPT_KEY. The encrypted key is just a plain text string that needs to be protected in the same way as any sensitive information.

To learn more about the use of asymmetric keys, see https://docs.spring.io/spring-cloud-config/docs/3.0.2/reference/html/#_key_management.

Introducing the config server API

The config server exposes a REST API that can be used by its clients to retrieve their configuration. In this chapter, we will use the following endpoints in the API:

• /actuator: The standard actuator endpoint exposed by all microservices.As always, these should be used with care. They are very useful during development but must be locked down before being used in production.
• /encrypt and /decrypt: Endpoints for encrypting and decrypting sensitive information. These must also be locked down before being used in production.
• /{microservice}/{profile}: Returns the configuration for the specified microservice and the specified Spring profile.

We will see some sample uses for the API when we try out the config server.

Setting up a config server

Setting up a config server on the basis of the decisions discussed is straightforward:

1. Create a Spring Boot project using Spring Initializr, as described in Chapter 3, Creating a Set of Cooperating Microservices. Refer to the Using Spring Initializr to generate skeleton code section.
2. Add the dependencies, spring-cloud-config-server and spring-boot-starter-security, to the Gradle build file, build.gradle.
3. Add the annotation @EnableConfigServer to the application class, ConfigServerApplication:

   @EnableConfigServer
   @SpringBootApplication
   public class ConfigServerApplication {
   

4. Add the configuration for the config server to the default property file, application.yml:

   server.port: 8888
   
   spring.cloud.config.server.native.searchLocations: file:${PWD}/config-repo
   
   management.endpoint.health.show-details: "ALWAYS"
   management.endpoints.web.exposure.include: "*"
   
   logging:
     level:
       root: info
   
   ---
   spring.config.activate.on-profile: docker
   spring.cloud.config.server.native.searchLocations: file:/config-repo
   

   The most important configuration is to specify where to find the configuration repository, indicated using the spring.cloud.config.server.native.searchLocations property.

5. Add a routing rule to the edge server to make the API of the config server accessible from outside the microservice landscape.
6. Add a Dockerfile and a definition of the config server to the three Docker Compose files.
7. Externalize sensitive configuration parameters to the standard Docker Compose environment file, .env. The parameters are described below, in the Configuring the config server for use with Docker section.
8. Add the config server to the common build file, settings.gradle:

   include ':spring-cloud:config-server'
   

The source code for the Spring Cloud Configuration server can be found in $BOOK_HOME/Chapter12/spring-cloud/config-server.

Now, let's look into how to set up the routing rule referred to in step 5 and how to configure the config server added in Docker Compose, as described in steps 6 and 7.

Setting up a routing rule in the edge server

To be able to access the API of the config server from outside the microservice landscape, we add a routing rule to the edge server. All requests to the edge server that begin with /config will be routed to the config server with the following routing rule:

 - id: config-server
   uri: http://${app.config-server}:8888
  predicates:
  - Path=/config/**
  filters:
  - RewritePath=/config/(?<segment>.*), /$\{segment}

The RewritePath filter in the routing rule will remove the leading part, /config, from the incoming URL before it sends it to the config server.

The edge server is also configured to permit all requests to the config server, delegating the security checks to the config server. The following line is added to the SecurityConfig class in the edge server:

  .pathMatchers("/config/**").permitAll()

With this routing rule in place, we can use the API of the config server; for example, run the following command to ask for the configuration of the product service when it uses the docker Spring profile:

curl https://dev-usr:dev-pwd@localhost:8443/config/product/docker -ks | jq

We will run this command when we try out the config server later on.

Configuring the config server for use with Docker

The Dockerfile of the config server looks the same as for the other microservices, except for the fact that it exposes port 8888 instead of port 8080.

When it comes to adding the config server to the Docker Compose files, it looks a bit different from what we have seen for the other microservices:

config-server:
  build: spring-cloud/config-server
  mem_limit: 512m
  environment:
    - SPRING_PROFILES_ACTIVE=docker,native
    - ENCRYPT_KEY=${CONFIG_SERVER_ENCRYPT_KEY}
    - SPRING_SECURITY_USER_NAME=${CONFIG_SERVER_USR}
    - SPRING_SECURITY_USER_PASSWORD=${CONFIG_SERVER_PWD}
  volumes:
    - $PWD/config-repo:/config-repo

Here are the explanations for the preceding source code:

1. The Spring profile, native, is added to signal to the config server that the config repository is based on local files
2. The environment variable ENCRYPT_KEY is used to specify the symmetric encryption key that will be used by the config server to encrypt and decrypt sensitive configuration information
3. The environment variables SPRING_SECURITY_USER_NAME and SPRING_SECURITY_USER_PASSWORD are used to specify the credentials to be used for protecting the APIs using basic HTTP authentication
4. The volume declaration will make the config-repo folder accessible in the Docker container at /config-repo

The values of the three preceding environment variables, marked in the Docker Compose file with ${...}, are fetched by Docker Compose from the .env file:

CONFIG_SERVER_ENCRYPT_KEY=my-very-secure-encrypt-key
CONFIG_SERVER_USR=dev-usr
CONFIG_SERVER_PWD=dev-pwd

The information stored in the .env file, that is, the username, password, and encryption key, is sensitive and must be protected if used for something other than development and testing. Also, note that losing the encryption key will lead to a situation where the encrypted information in the config repository cannot be decrypted!

Configuring clients of a config server

To be able to get their configurations from the config server, our microservices need to be updated. This can be done with the following steps:

1. Add the spring-cloud-starter-config and spring-retry dependencies to the Gradle build file, build.gradle.
2. Move the configuration file, application.yml, to the config repository and rename it with the name of the client as specified by the property spring.application.name.
3. Add a new application.yml file to the src/main/resources folder. This file will be used to hold the configuration required to connect to the config server. Refer to the following Configuring connection information section for an explanation of its content.
4. Add credentials for accessing the config server to the Docker Compose files, for example, the product service:

   product:
     environment:
    - CONFIG_SERVER_USR=${CONFIG_SERVER_USR}
    - CONFIG_SERVER_PWD=${CONFIG_SERVER_PWD}
   

5. Disable the use of the config server when running Spring Boot-based automated tests. This is done by adding spring.cloud.config.enabled=false to the @DataMongoTest, @DataJpaTest, and @SpringBootTest annotations. They look like:

   @DataMongoTest(properties = {"spring.cloud.config.enabled=false"})
   
   @DataJpaTest(properties = {"spring.cloud.config.enabled=false"})
   
   @SpringBootTest(webEnvironment=RANDOM_PORT, properties = {"eureka.client.enabled=false", "spring.cloud.config.enabled=false"})
   

Starting with Spring Boot 2.4.0, the processing of multiple property files has changed rather radically. The most important changes, applied in this book, are:

• The order in which property files are loaded. Starting with Spring Boot 2.4.0, they are loaded in the order that they're defined.
• How property override works. Starting with Spring Boot 2.4.0, properties declared lower in a file will override those higher up.
• A new mechanism for loading additional property files, for example, property files from a config server, has been added. Starting with Spring Boot 2.4.0, the property spring.config.import can be used as a common mechanism for loading additional property files.

For more information and the reasons for making these changes, see https://spring.io/blog/2020/08/14/config-file-processing-in-spring-boot-2-4.

Spring Cloud Config v3.0.0, included in Spring Cloud 2020.0.0, supports the new mechanism for loading property files in Spring Boot 2.4.0. This is now the default mechanism for importing property files from a config repository. This means that the Spring Cloud Config-specific bootstrap.yml files are replaced by standard application.yml files, using a spring.config.import property to specify that additional configuration files will be imported from a config server. It is still possible to use the legacy bootstrap way of importing property files; for details, see https://docs.spring.io/spring-cloud-config/docs/3.0.2/reference/html/#config-data-import.

Configuring connection information

As mentioned previously, the src/main/resources/application.yml file now holds the client configuration that is required to connect to the config server. This file has the same content for all clients of the config server, except for the application name as specified by the spring.application.name property (in the following example, set to product):

spring.config.import: "configserver:"

spring:
  application.name: product
  cloud.config:
    failFast: true
    retry:
      initialInterval: 3000
      multiplier: 1.3
      maxInterval: 10000
      maxAttempts: 20
    uri: http://localhost:8888
    username: ${CONFIG_SERVER_USR}
    password: ${CONFIG_SERVER_PWD}

---
spring.config.activate.on-profile: docker

spring.cloud.config.uri: http://config-server:8888

This configuration will make the client do the following:

1. Connect to the config server using the http://localhost:8888 URL when it runs outside Docker, and using the http://config-server:8888 URL when running in a Docker container
2. Use HTTP Basic authentication, based on the value of the CONFIG_SERVER_USR and CONFIG_SERVER_PWD properties, as the client's username and password
3. Try to reconnect to the config server during startup up to 20 times, if required
4. If the connection attempt fails, the client will initially wait for 3 seconds before trying to reconnect
5. The wait time for subsequent retries will increase by a factor of 1.3
6. The maximum wait time between connection attempts will be 10 seconds
7. If the client can't connect to the config server after 20 attempts, its startup will fail

This configuration is generally good for resilience against temporary connectivity problems with the config server. It is especially useful when the whole landscape of microservices and its config server are started up at once, for example, when using the docker-compose up command. In this scenario, many of the clients will be trying to connect to the config server before it is ready, and the retry logic will make the clients connect to the config server successfully once it is up and running.

Structuring the configuration repository

After moving the configuration files from each client's source code to the configuration repository, we will have some common configuration in many of the configuration files, for example, for the configuration of actuator endpoints and how to connect to Eureka, RabbitMQ, and Kafka. The common parts have been placed in a common configuration file named application.yml. This file is shared by all clients. The configuration repository contains the following files:

config-repo/
├── application.yml
├── auth-server.yml
├── eureka-server.yml
├── gateway.yml
├── product-composite.yml
├── product.yml
├── recommendation.yml
└── review.yml

The configuration repository can be found in $BOOK_HOME/Chapter12/config-repo.

Trying out the Spring Cloud Configuration server

Now it is time to try out the config server:

1. First, we will build from source and run the test script to ensure that everything fits together
2. Next, we will try out the config server API to retrieve the configuration for our microservices
3. Finally, we will see how we can encrypt and decrypt sensitive information, for example, passwords

Building and running automated tests

So now we build and run verification tests of the system landscape, as follows:

1. First, build the Docker images with the following commands:

   cd $BOOK_HOME/Chapter12
   ./gradlew build && docker-compose build
   

2. Next, start the system landscape in Docker and run the usual tests with the following command:

   ./test-em-all.bash start
   

Getting the configuration using the config server API

As already described previously, we can reach the API of the config server through the edge server by using the URL prefix, /config. We also have to supply credentials as specified in the .env file for HTTP Basic authentication. For example, to retrieve the configuration used for the product service when it runs as a Docker container, that is, having activated the Spring profile docker, run the following command:

curl https://dev-usr:dev-pwd@localhost:8443/config/product/docker -ks | jq .

Expect a response with the following structure (many of the properties in the response are replaced by ... to increase readability):

{
  "name": "product",
  "profiles": [
    "docker"
  ],
  ...
  "propertySources": [
    {
      "name": "...file [/config-repo/product.yml]...",
      "source": {
        "spring.config.activate.on-profile": "docker",
        "server.port": 8080,
        ...
      }
    },
    {
      "name": "...file [/config-repo/product.yml]...",
      "source": {
        "server.port": 7001,
        ...
      }
    },
    {
      "name": "...file [/config-repo/application.yml]...",
      "source": {
        "spring.config.activate.on-profile": "docker",
        ...
      }
    },
    {
      "name": "...file [/config-repo/application.yml]...",
      "source": {
        ...
        "app.eureka-password": "p",
        "spring.rabbitmq.password": "guest"
      }
    }
  ]
}

The explanations for this response are as follows:

• The response contains properties from a number of property sources, one per property file and Spring profile that matched the API request. The property sources are returned in priority order; if a property is specified in multiple property sources, the first property in the response takes precedence. The preceding sample response contains the following property sources, in the following priority order:
  • /config-repo/product.yml, for the docker Spring profile
  • /config-repo/product.yml, for the default Spring profile
  • /config-repo/application.yml, for the docker Spring profile
  • /config-repo/application.yml, for the default Spring profile

  For example, the port used will be 8080 and not 7001, since "server.port": 8080 is specified before "server.port": 7001 in the preceding response.

• Sensitive information, such as the passwords to Eureka and RabbitMQ, are returned in plain text, for example, "p" and "guest", but they are encrypted on disk. In the configuration file, application.yml, they are specified as follows:

  app:
    eureka-password:
  '{cipher}bf298f6d5f878b342f9e44bec08cb9ac00b4ce57e98316f030194a225 fac89fb'
  spring.rabbitmq:
    password: '{cipher}17fcf0ae5b8c5cf87de6875b699be4a1746dd493a99d926c7a26a68c422117ef'
  

Encrypting and decrypting sensitive information

Information can be encrypted and decrypted using the /encrypt and /decrypt endpoints exposed by the config server. The /encrypt endpoint can be used to create encrypted values to be placed in the property file in the config repository. Refer to the example in the previous section, where the passwords to Eureka and RabbitMQ are stored encrypted on disk. The /decrypt endpoint can be used to verify encrypted information that is stored on disk in the config repository.

To encrypt the hello world string, run the following command:

curl -k https://dev-usr:dev-pwd@localhost:8443/config/encrypt --data-urlencode "hello world"

It is important to use the --data-urlencode flag when using curl to call the /encrypt endpoint, to ensure the correct handling of special characters such as '+'.

Expect a response along the lines of the following:

Figure 12.2: An encrypted value of a configuration parameter

To decrypt the encrypted value, run the following command:

curl -k https://dev-usr:dev-pwd@localhost:8443/config/decrypt -d 9eca39e823957f37f0f0f4d8b2c6c46cd49ef461d1cab20c65710823a8b412ce

Expect the hello world string as the response:

Figure 12.3: A decrypted value of a configuration parameter

If you want to use an encrypted value in a configuration file, you need to prefix it with {cipher} and wrap it in ''. For example, to store the encrypted version of hello world, add the following line in a YAML-based configuration file:

my-secret:  '{cipher}9eca39e823957f37f0f0f4d8b2c6c46cd49ef461d1cab20c65710823a8b412ce'

When the config server detects values in the format '{cipher}...', it tries to decrypt them using its encryption key before sending them to a client.

These tests conclude the chapter on centralized configuration. Wrap it up by shutting down the system landscape:

docker-compose down

Summary

In this chapter, we have seen how we can use the Spring Cloud Configuration Server to centralize managing the configuration of our microservices. We can place the configuration files in a common configuration repository and share common configurations in a single configuration file, while keeping microservice-specific configuration in microservice-specific configuration files. The microservices have been updated to retrieve their configuration from the config server at startup and are configured to handle temporary outages while retrieving their configuration from the config server.

The config server can protect configuration information by requiring authenticated usage of its API with HTTP Basic authentication and can prevent eavesdropping by exposing its API externally through the edge server that uses HTTPS. To prevent intruders who obtained access to the configuration files on disk from gaining access to sensitive information such as passwords, we can use the config server /encrypt endpoint to encrypt the information and store it encrypted on disk.

While exposing the APIs from the config server externally is useful during development, they should be locked down before use in production.

In the next chapter, we will learn how we can use Resilience4j to mitigate the potential drawbacks of overusing synchronous communication between microservices.

Questions

1. What API call can we expect from a review service to the config server during startup to retrieve its configuration?
2. The review service was started up using the following command: docker compose up -d.

   What configuration information should we expect back from an API call to the config server using the following command?

   curl https://dev-usr:dev-pwd@localhost:8443/config/application/default -ks | jq 
   

3. What types of repository backend does Spring Cloud Config support?
4. How can we encrypt sensitive information on disk using Spring Cloud Config?
5. How can we protect the config server API from misuse?
6. Mention some pros and cons for clients that first connect to the config server as opposed to those that first connect to the discovery server.