Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases now! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Continuous Delivery with Docker and Jenkins
Continuous Delivery with Docker and Jenkins

Continuous Delivery with Docker and Jenkins: Delivering software at scale

eBook
$27.98 $39.99
Paperback
$48.99
Subscription
Free Trial
Renews at $19.99p/m

What do you get with Print?

Product feature icon Instant access to your digital eBook copy whilst your Print order is Shipped
Product feature icon Paperback book shipped to your preferred address
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
Table of content icon View table of contents Preview book icon Preview Book

Continuous Delivery with Docker and Jenkins

Introducing Continuous Delivery

The common problem faced by most developers is how to release the implemented code quickly and safely. The delivery process used traditionally is, however, a source of pitfalls and usually leads to the disappointment of both developers and clients. This chapter presents the idea of the Continuous Delivery approach and provides the context for the rest of the book.

This chapter covers the following points:

  • Introducing the traditional delivery process and its drawbacks
  • Describing the idea of Continuous Delivery and the benefits it brings
  • Comparing how different companies deliver their software
  • Explaining the automated deployment pipeline and its phases
  • Classifying different types of tests and their place in the process
  • Pointing out the prerequisites to the successful Continuous Delivery process
  • Presenting tools that will be used throughout the book
  • Showing the complete system that will be built throughout the book

What is Continuous Delivery?

The most accurate definition of the Continuous Delivery is stated by Jez Humble and reads as follows: "Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes, and experiments—into production, or into the hands of users, safely and quickly in a sustainable way." That definition covers the key points.

To understand it better, let's imagine a scenario. You are responsible for the product, let's say, the email client application. Users come to you with a new requirement—they want to sort emails by size. You decide that the development will take around one week. When can the user expect to use the feature? Usually, after the development is done, you hand over the completed feature first to the QA team and then to the operations team, which takes additional time ranging from days to months. Therefore, even though the development took only one week, the user receives it in a couple of months! The Continuous Delivery approach addresses that issue by automating manual tasks so that the user could receive a new feature as soon as it's implemented.

To present better what to automate and how, let's start by describing the delivery process that is currently used for most software systems.

The traditional delivery process

The traditional delivery process, as the name suggests, has been in place for many years now and is implemented in most IT companies. Let's define how it works and comment on its shortcomings.

Introducing the traditional delivery process

Any delivery process begins with the requirements defined by a customer and ends up with the release on the production. The differences are in between. Traditionally, it looks as presented in the following release cycle diagram:

The release cycle starts with the requirements provided by the Product Owner, who represents the Customer (stakeholders). Then there are three phases, during which the work is passed between different teams:

  • Development: Here, the developers (sometimes together with business analysts) work on the product. They often use Agile techniques (Scrum or Kanban) to increase the development velocity and to improve the communication with the client. Demo sessions are organized to obtain a customer's quick feedback. All good development techniques (like test-driven development or extreme programming practices) are welcome. After the implementation is completed, the code is passed to the QA team.
  • Quality Assurance: This phase is usually called User Acceptance Testing (UAT) and it requires a code freeze on the trunk codebase, so that no new development would break the tests. The QA team performs a suite of Integration Testing, Acceptance Testing, and Non-functional Testing (performance, recovery, security, and so on). Any bug that is detected goes back to the development team, so developers usually also have their hands full of work. After the UAT phase is completed, the QA team approves the features that are planned for the next release.
  • Operations: The last phase, usually the shortest one, means passing the code to the Operations team, so that they can perform the release and monitor the production. If anything goes wrong, they contact developers to help with the production system.

The length of the release cycle depends on the system and the organization, but it usually ranges from a week to a few months. The longest I've heard about was one year.The longest I worked with was quarterly-based and each part took as follows: development-1.5 months, UAT-1 month and 3 weeks, release (and strict production monitoring)-1 week.

The traditional delivery process is widely used in the IT industry and it's probably not the first time you've read about such an approach. Nevertheless, it has a number of drawbacks. Let's look at them explicity to understand why we need to strive for something better.

Shortcomings of the traditional delivery process

The most significant shortcomings of the traditional delivery process include the following:

  • Slow delivery: Here, the customer receives the product long after the requirements were specified. It results in the unsatisfactory time to market and delays of the customer's feedback.
  • Long feedback cycle: The feedback cycle is not only related to customers, but also to developers. Imagine that you accidentally created a bug and you learn about it during the UAT phase. How long does it take to fix something you worked on two months ago? Even minor bugs can consume weeks.
  • Lack of automation: Rare releases don't encourage the automation, which leads to unpredictable releases.
  • Risky hotfixes: Hotfixes can't usually wait for the full UAT phase, so they tend to be tested differently (the UAT phase is shortened) or not tested at all.
  • Stress: Unpredictable releases are stressful for the operations team. What's more, the release cycle is usually tightly scheduled which imposes an additional stress on developers and testers.
  • Poor communication: Work passed from one team to another represents the waterfall approach, in which people start to care only about their part, rather than the complete product. In case anything goes wrong, that usually leads to the blaming game instead of cooperation.
  • Shared responsibility: No team takes the responsibility for the product from A to Z. For developers: "done" means that requirements are implemented. For testers: "done" means that the code is tested. For operations: "done" means that the code is released.
  • Lower job satisfaction: Each phase is interesting for a different team, but other teams need to support the process. For example, the development phase is interesting for developers but, during two other phases, they still need to fix bugs and support the release, which usually is not interesting for them at all.

These drawbacks represent just a tip of the iceberg of the challenges related to the traditional delivery process. You may already feel that there must be a better way to develop the software and this better way is, obviously, the Continuous Delivery approach.

Benefits of Continuous Delivery

“How long would it take your organization to deploy a change that involves just one single line of code? Do you do this on a repeatable, reliable basis?" These are the famous questions from Mary and Tom Poppendieck (authors of Implementing Lean Software Development), which have been quoted many times by Jez Humble and other authors. Actually, the answer to these questions is the only valid measurement of the health of your delivery process.

To be able to deliver continuously, and not to spend a fortune on the army of operations teams working 24/7, we need automation. That is why, in short, Continuous Delivery is all about changing each phase of the traditional delivery process into a sequence of scripts, called the automated deployment pipeline or the Continuous Delivery pipeline. Then, if no manual steps are required, we can run the process after every code change and, therefore, deliver the product continuously to the users.

Continuous Delivery lets us get rid of the tedious release cycle and, therefore, brings the following benefits:

  • Fast delivery: Time to market is significantly reduced as customers can use the product as soon as the development is completed. Remember, the software delivers no revenue until it is in the hands of its users.
  • Fast feedback cycle: Imagine you created a bug in the code, which goes into the production the same day. How much time does it take to fix something you worked on the same day? Probably not much. This, together with the quick rollback strategy, is the best way to keep the production stable.
  • Low-risk releases: If you release on a daily basis, then the process becomes repeatable and therefore much safer. As the saying goes, "If it hurts, do it more often."
  • Flexible release options: In case you need to release immediately, everything is already prepared, so there is no additional time/cost associated with the release decision.

Needless to say, we could achieve all the benefits simply by eliminating all delivery phases and proceeding with the development directly on the production. It would, however, cause the quality to decline. Actually, the whole difficulty of introducing Continuous Delivery is the concern that the quality would decrease together with eliminating manual steps. In this book, we will show how to approach it in a safe manner and explain why, contrary to common beliefs, the products delivered continuously have fewer bugs and are better adjusted to the customer's needs.

Success stories

My favorite story on Continuous Delivery was told by Rolf Russell at one of his talks. It goes as follows. In 2005, Yahoo acquired Flickr and it was a clash of two cultures in the developer's world. Flickr, by that time, was a company with the start-up approach in mind. Yahoo, on the contrary, was a huge corporation with strict rules and the safety-first attitude. Their release processes differed a lot. While Yahoo used the traditional delivery process, Flickr released many times a day. Every change implemented by developers went on the production the same day. They even had a footer at the bottom of their page showing the time of the last release and the avatars of the developers who did the changes.

Yahoo deployed rarely and each release brought a lot of changes well tested and prepared. Flickr worked in very small chunks, each feature was divided into small incremental parts and each part was deployed quickly to the production. The difference is presented in the following diagram:

You can imagine what happened when the developers from two companies met. Yahoo obviously treated Flickr's colleagues as junior irresponsible developers, "a bunch of software cowboys who don't know what they are doing." So, the first thing they wanted to change was to add a QA team and the UAT phase into Flickr's delivery process. Before they applied the change, however, Flickr's developers had only one wish. They asked to evaluate the most reliable products in the whole Yahoo company. What a surprise when it happened that of all the software in Yahoo, Flickr had the lowest downtime. The Yahoo team didn't understand it at first, but let Flickr stay with their current process anyway. After all, they were engineers, so the evaluation result was conclusive. Only after some time, they realized that the Continuous Delivery process can be beneficial for all products in Yahoo and they started to gradually introduce it everywhere.

The most important question of the story remains-how was it possible that Flickr was the most reliable system? Actually, the reason for that fact was what we already mentioned in the previous sections. A release is less risky if:

  • The delta of code changes is small
  • The process is repeatable

That is why, even though the release itself is a difficult activity, it is much safer when done frequently.

The story of Yahoo and Flickr is only an example of many successful companies for which the Continuous Delivery process proved to be right. Some of them even proudly share details from their systems, as follows:

  • Amazon: In 2011, they announced reaching 11.6 seconds (on average) between deployments
  • Facebook: In 2013, they announced deployment of code changes twice a day
  • HubSpot: In 2013, they announced deployment 300 times a day
  • Atlassian: In 2016, they published a survey stating that 65% of their customers practice continuous delivery
You can read more about the research on the Continuous Delivery process and individual case studies at https://continuousdelivery.com/evidence-case-studies/.

Keep in mind that the statistics get better every day. However, even without any numbers, just imagine a world in which every line of code you implement goes safely into the production. Clients can react quickly and adjust their requirements, developers are happy because they don't have to solve that many bugs, managers are satisfied because they always know what is the current state of work. After all, remember, the only true measure of progress is the released software.

The automated deployment pipeline

We already know what the Continuous Delivery process is and why we use it. In this section, we describe how to implement it.

Let's start by emphasizing that each phase in the traditional delivery process is important. Otherwise, it would never have been created in the first place. No one wants to deliver software without testing it first! The role of the UAT phase is to detect bugs and to ensure that what developers created is what the customer wanted. The same applies to the operations team—the software must be configured, deployed to the production, and monitored. That's out of the question. So, how do we automate the process so that we preserve all the phases? That is the role of the automated deployment pipeline, which consists of three stages as presented in the following diagram:

The automated deployment pipeline is a sequence of scripts that is executed after every code change committed to the repository. If the process is successful, it ends up with the deployment to the production environment.

Each step corresponds to a phase in the traditional delivery process as follows:

  • Continuous Integration: This checks to make sure that the code written by different developers integrates together
  • Automated Acceptance Testing: This replaces the manual QA phase and checks if the features implemented by developers meet the client's requirements
  • Configuration Management: This replaces the manual operations phase-configures the environment and deploys the software

Let's take a deeper look at each phase to understand what is its responsibility and what steps it includes.

Continuous Integration

The Continuous Integration phase provides the first feedback to developers. It checks out the code from the repository, compiles it, runs unit tests, and verifies the code quality. If any step fails, the pipeline execution is stopped and the first thing the developers should do is fix the Continuous Integration build. The essential aspect of the phase is time; it must be executed in a timely manner. For example, if this phase took an hour to complete then the developers would commit the code faster, which would result in the constantly failing pipeline.

The Continuous Integration pipeline is usually the starting point. Setting it up is simple because everything is done within the development team and no agreement with the QA and operations teams is necessary.

Automated acceptance testing

The automated acceptance testing phase is a suite of tests written together with the client (and QAs) that is supposed to replace the manual UAT stage. It acts as a quality gate to decide whether a product is ready for the release or not. If any of the acceptance tests fail, then the pipeline execution is stopped and no further steps are run. It prevents movement to the Configuration Management phase and therefore the release.

The whole idea of automating the acceptance phase is to build the quality into the product instead of verifying it later. In other words, when a developer completes the implementation, the software is delivered already together with acceptance tests which verify that the software is what the client wanted. That is a large shift in thinking about testing software. There is no longer a single person (or team) who approves the release, but everything depends on passing the acceptance test suite. That is why creating this phase is usually the most difficult part of the Continuous Delivery process. It requires a close cooperation with the client and creating tests at the beginning (not at the end) of the process.

Introducing automated acceptance tests is especially challenging in the case of legacy systems. We describe more on that topic in Chapter 9, Advanced Continuous Delivery.

There is usually a lot of confusion about the types of tests and their place in the Continuous Delivery process. It's also often unclear how to automate each type, what should be the coverage, and what should be the role of the QA team in the whole development process. Let's clarify it using the Agile testing matrix and the testing pyramid.

The Agile testing matrix

Brian Marick, in a series of his blog posts, made a classification of software tests in a form of the so-called agile testing matrix. It places tests in two dimensions: business or technology facing and support programmers or critique the product. Let's have a look at that classification:

Let's comment briefly on each type of test:

  • Acceptance Testing (automated): These are tests that represent functional requirements seen from the business perspective. They are written in the form of stories or examples by clients and developers to agree on how the software should work.
  • Unit Testing (automated): These are tests that help developers to provide the high-quality software and minimize the number of bugs.
  • Exploratory Testing (manual): This is the manual black-box testing, which tries to break or improve the system.
  • Non-functional Testing (automated): These are tests that represent system properties related to the performance, scalability, security, and so on.

This classification answers one of the most important questions about the Continuous Delivery process: what is the role of a QA in the process?

Manual QAs perform the exploratory testing, so they play with the system, try to break it, ask questions, think about improvements. Automation QAs help with nonfunctional and acceptance testing, for example, they write code to support load testing. In general, QAs don't have their special place in the delivery process, but rather a role in the development team.

In the automated Continuous Delivery process, there is no longer a place for manual QAs who perform repetitive tasks.

You may look at the classification and wonder why you see no integration tests there. Where are they up to Brian Marick and where to put them in the Continuous Delivery pipeline?

To explain it well, we first need to mention that the meaning of an integration test differs depending on the context. For (micro) service architecture, they usually mean exactly the same as the acceptance testing, as services are small and need nothing more than unit and acceptance tests. If you build a modular application, then by integration tests we usually mean component tests that bind multiple modules (but not the whole application) and test them together. In that case, integration tests place themselves somewhere between acceptance and unit tests. They are written in a similar way as acceptance tests, but are usually more technical and require mocking not only external services, but also internal modules. Integration tests, similar to unit tests, represent the "code" point of view, while acceptance tests represent the "user" point of view. Concerning the Continuous Delivery pipeline, integration tests are simply implemented as a separate phase in the process.

The testing pyramid

The previous section explained what each test type represents in the process, but mentioned nothing about how many tests we should develop. So, what should be the code coverage in case of unit testing? What about acceptance testing?

To answer these questions, Mike Cohn, in his book Succeeding with Agile: Software Development Using Scrum, created a so-called testing pyramid. Let's look at the diagram to understand it well.

When we move up the pyramid, the tests become slower and more expensive to create. They often require touching user interface and hiring a separate test automation team. That is why acceptance tests should not target 100% coverage. On the contrary, they should be feature-oriented and verify only selected test scenarios. Otherwise, we would spend a fortune on the test development and maintenance, and our Continuous Delivery pipeline build would take ages to execute.

The case is different at the bottom of the pyramid. Unit tests are cheap and fast, so we should strive for 100% code coverage. They are written by developers and providing them should be a standard procedure for any mature team.

I hope that the agile testing matrix and the testing pyramid clarified the role and the importance of acceptance testing.

Let's move to the last phase of the Continuous Delivery process, configuration management.

Configuration management

The configuration management phase is responsible for tracking and controlling changes in the software and its environment. It concerns taking care of preparing and installing the necessary tools, scaling the number of service instances and their distribution, infrastructure inventory, and all tasks related to the application deployment.

Configuration management is a solution to the problems posed by manually deploying and configuring applications on the production. Such common practice results in an issue whereby we no longer knows where each service is running and with what properties. Configuration management tools (such as Ansible, Chef, or Puppet) enable storing configuration files in the version control system and tracking every change that was made on the production servers.

An additional effort to replace manual tasks of the operations team is to take care of application monitoring. That is usually done by streaming logs and metrics of the running systems to a common dashboard, which is monitored by developers (or the DevOps team, as explained in the next section).

Prerequisites to Continuous Delivery

The rest of the book is dedicated to technical details on how to implement a successful Continuous Delivery pipeline. The success of the process, however, depends not only on the tools we present throughout the book. In this section, we take a holistic look at the whole process and define the Continuous Delivery requirements in three areas:

  • Your organization's structure and its impact on the development process
  • Your products and their technical details
  • Your development team and the practices you use

Organizational prerequisites

The way your organization works has a high impact on the success of introducing the Continuous Delivery process. It's a bit similar to introducing Scrum. Many organizations would like to use the Agile process, but they don't change their culture. You can't use Scrum in your development team unless the organization's structure is adjusted to that. For example, you need a product owner, stakeholders, and management that understands that no requirement changes are possible during the sprint. Otherwise, even with good will, you won't make it. The same applies to the Continuous Delivery process; it requires an adjustment of how the organization is structured. Let's have a look at three aspects: the DevOps culture, a client in the process, and business decisions.

DevOps culture

A long time ago, when software was written by individuals or microteams, there was no clear separation between the development, quality assurance, and operations. A person developed the code, tested it, and then put it into the production. If anything went wrong, the same person investigated the issue, fixed it, and redeployed to the production. The way the development is organized now changed gradually, when systems became larger and development teams grew. Then, engineers started to become specialized in one area. That made perfect sense, because specialization caused a boost in the productivity. However, the side effect was the communication overhead. It is especially visible if developers, QAs, and operations are under separate departments in the organization, sit in different buildings, or are outsourced to different countries. Such organization structure is no good for the Continuous Delivery process. We need something better, we need to adapt the so-called DevOps culture.

DevOps culture means, in a sense, coming back to the roots. A single person or a team is responsible for all three areas, as presented in the following diagram:

The reason why it's possible to move to the DevOps model without losing on the productivity is the automation. Most of the tasks related to the quality assurance and operations are moved to the automated delivery pipeline and can be therefore managed by the development team.

A DevOps team doesn't necessarily need to consist only of developers. A very common scenario in many organization's under transformation is to create teams with four developers, one QA, and one person from operations. They need, however, to work closely together (sit in one area, have stand-ups together, work on the same product).

The culture of small DevOps teams affects the software architecture. Functional requirements have to be well separated into (micro) services or modules, so that each team can take care of an independent part.

The impact of the organization's structure on the software architecture was already observed in 1967 and formulated as Conway's Law: "Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure."

Client in the process

The role of a client (or a product owner) slightly changes during the Continuous Delivery adoption. Traditionally, clients are involved in defining requirements, answering questions from developers, attending demos, and taking part in the UAT phase to agree if what was built is what they had in mind.

In Continuous Delivery, there is no UAT, and a client is essential in the process of writing acceptance tests. For some clients, who already wrote their requirements in a testable manner, it is not a big shift. For the others, it means a change in a way of thinking to make requirements more technical-oriented.

In the Agile environment, some teams don't even accept user stories (requirements) without acceptance tests attached. Such techniques, even though they may sound too strict, often lead to better development productivity.

Business decisions

In most companies, the business has an impact on the release schedule. After all, the decision what features are delivered, and when, is related to different departments of the company (for example, marketing) and can be strategic for the enterprise. That is why the release scheduling has to be reapproached and discussed between the business and the development teams.

Obviously, there are techniques such as feature toggles or manual pipeline steps, which help with releasing features at the specified time. We will describe them later in the book. To be precise, the term Continuous Delivery is not the same as Continuous Deployment. The former means that each commit to the repository is automatically released to the production. Continuous Delivery is less strict and means that each commit ends up with a release candidate, so it allows the last step (release to the production) to be manual.

In the rest of the book, we will use the terms Continuous Delivery and Continuous Deployment interchangeably.

Technical and development prerequisites

From the technical side, there are a few requirements to keep in mind. We will discuss them throughout the book, so let's only mention them here without going into detail:

  • Automated build, test, package, and deploy operations: All operations need to be possible to automate. If we deal with the system that is non-automatable, for example, due to security reasons or its complexity, then it's impossible to create a fully automated delivery pipeline.
  • Quick pipeline execution: The pipeline must be executed in a timely manner, preferably in 5-15 minutes. If our pipeline execution takes hours or days, then it won't be possible to run it after every commit to the repository.
  • Quick failure recovery: A possibility of the quick rollback or system recovery is a must. Otherwise, we risk the production health due to frequent releases.
  • Zero-downtime deployment: The deployment cannot have any downtime since we release many times a day.
  • Trunk-based development: Developers must check in regularly into one master branch. Otherwise, if everyone develops in their own branches, the integration is rare and therefore the releases are rare, which is exactly the opposite of what we want to achieve.

We will write more on these prerequisites and how to address them throughout the book. Keeping that in mind, let's move to the last section of this chapter and introduce what system we plan to build in this book and what tools we will use for that purpose.

Building the Continuous Delivery process

We introduced the idea, benefits, and prerequisites with regards to the Continuous Delivery process. In this section, we describe the tools that will be used throughout the book and their place in the complete system.

If you're interested more in the idea of the Continuous Delivery process, then have a look at an excellent book by Jez Humble and David Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation.

Introducing tools

First of all, the specific tool is always less important than understanding its role in the process. In other words, any tool can be replaced with another one which plays the same role. For example, Jenkins can be replaced with Atlassian Bamboo and Chief can be used instead of Ansible. That is why each chapter begins with the general description of why such a tool is necessary and what its role is in the whole process. Then, the exact tool is described with comparison to its substitutes. That form gives you the flexibility to choose the right one for your environment.

Another approach could be to describe the Continuous Delivery process on the level of ideas; however, I strongly believe that giving an exact example with the code extract, something that readers can run by themselves, results in a much better understanding of the concept.

There are two ways to read this book. The first is to read and understand the concepts of the Continuous Delivery process. The second is to create your own environment and execute all scripts while reading to understand the details.

Let's have a quick look at the tools we will use throughout the book. In this section, however, it is only a brief introduction of each technology and much more detail is presented as this book goes on.

Docker ecosystem

Docker, as the clear leader of the containerization movement, has dominated the software industry in the recent years. It allows the packaging of an application in the environment-agnostic image and therefore treats servers as a farm of resources, rather than machines that must be configured for each application. Docker was a clear choice for this book because it perfectly fits the (micro) service world and the Continuous Delivery process.

Together with Docker comes additional technologies, which are as follows:

  • Docker Hub: This is a registry for Docker images
  • Docker Compose: This is a tool to define multicontainer Docker applications
  • Docker Swarm: This is a clustering and scheduling tool

Jenkins

Jenkins is by far the most popular automation server on the market. It helps to create Continuous Integration and Continuous Delivery pipelines and, in general, any other automated sequence of scripts. Highly plugin-oriented, it has a great community which constantly extends it with new features. What's more, it allows to write the pipeline as code and supports distributed build environments.

Ansible

Ansible is an automation tool that helps with software provisioning, configuration management, and application deployment. It is trending faster than any other configuration management engine and can soon overtake its two main competitors: Chef and Puppet. It uses agentless architecture and integrates smoothly with Docker.

GitHub

GitHub is definitely the number one of all hosted version control systems. It provides a very stable system, a great web-based UI, and a free service for public repositories. Having said that, any source control management service or tool will work with Continuous Delivery, no matter if it's in the cloud or self-hosted and if it's based on Git, SVN, Mercurial, or any other tool.

Java/Spring Boot/Gradle

Java has been the most popular programming language for years. That is why it is being used for most code examples in this book. Together with Java, most companies develop with the Spring framework, so we used it to create a simple web service needed to explain some concepts. Gradle is used as a build tool. It's still less popular than Maven, however, trending much faster. As always, any programming language, framework, or build tool can be exchanged and the Continuous Delivery process would stay the same, so don't worry if your technology stack is different.

The other tools

Cucumber was chosen arbitrarily as the acceptance testing framework. Other similar solutions are Fitnesse and JBehave. For the database migration we use Flyway, but any other tool would do, for example, Liquibase.

Creating a complete Continuous Delivery system

You can look at how this book is organized from two perspectives.

The first one is based on the steps of the automated deployment pipeline. Each chapter takes you closer to the complete Continuous Delivery process. If you look at the names of the chapters, some of them are even named like the pipeline phases:

  • Continuous Integration pipeline
  • Automated acceptance testing
  • Configuration management with Ansible

The rest of the chapters give the introduction, summary, or additional information complementary to the process.

There is also a second perspective to the content of this book. Each chapter describes one piece of the environment, which in turn is well prepared for the Continuous Delivery process. In other words, the book presents, step by step, technology by technology, how to build a complete system. To help you get the feeling of what we plan to build throughout the book, let's now have a look at how the system will evolve in each chapter.

Don't worry if you don't understand the concepts and the terminology at this point. We explain everything from scratch in the corresponding chapters.

Introducing Docker

In Chapter 2, Introducing Docker, we start from the center of our system and build a working application packaged as a Docker image. The output of this chapter is presented in the following diagram:

A dockerized application (web service) is run as a container on a Docker Host and is reachable as it would run directly on the host machine. That is possible thanks to port forwarding (port publishing in the Docker's terminology).

Configuring Jenkins

In Chapter 3, Configuring Jenkins, we prepare the Jenkins environment. Thanks to the support of multiple agent (slave) nodes, it is able to handle the heavy concurrent load. The result is presented in the following diagram:

The Jenkins master accepts a build request, but the execution is started at one of the Jenkins Slave (agent) machines. Such an approach provides horizontal scaling of the Jenkins environment.

Continuous Integration Pipeline

In Chapter 4, Continuous Integration Pipeline, we show how to create the first phase of the Continuous Delivery pipeline, the commit stage. The output of this chapter is the system presented in the following diagram:

The application is a simple web service written in Java with the Spring Boot framework. Gradle is used as a build tool and GitHub as the source code repository. Every commit to GitHub automatically triggers the Jenkins build, which uses Gradle to compile Java code, run unit tests, and perform additional checks (code coverage, static code analysis, and so on). After the Jenkins build is completed, a notification is sent to the developers.

After this chapter, you will be able to create a complete Continuous Integration pipeline.

Automated acceptance testing

In Chapter 5, Automated Acceptance Testing, we finally merge the two technologies from the book title: Docker and Jenkins. It results in the system presented in the following diagram:

The additional elements in the diagram are related to the automated acceptance testing stage:

  • Docker Registry: After the Continuous Integration phase, the application is packaged first into a JAR file and then as a Docker image. That image is then pushed to the Docker Registry, which acts as a storage for dockerized applications.
  • Docker Host: Before performing the acceptance test suite, the application has to be started. Jenkins triggers a Docker Host machine to pull the dockerized application from the Docker Registry and starts it.
  • Docker Compose: If the complete application consists of more than one Docker container (for example, two web services: Application 1 using Application 2), then Docker Compose helps to run them together.
  • Cucumber: After the application is started on the Docker Host, Jenkins runs a suite of acceptance tests written in the Cucumber framework.

Configuration management with Ansible/Continuous Delivery pipeline

In the next two chapters, that is, Chapter 6, Configuration Management with Ansible and Chapter 7, Continuous Delivery Pipeline, we complete the Continuous Delivery pipeline. The output is the environment presented in the following diagram:

Ansible takes care of the environments and enables the deployment of the same applications on multiple machines. As a result, we deploy the application to the staging environment, run the acceptance testing suite, and finally release the application to the production environment, usually in many instances (on multiple Docker Host machines).

Clustering with Docker Swarm/Advanced Continuous Delivery

In Chapter 8, Clustering with Docker Swarm, we replace single hosts in each of the environments with clusters of machines. Chapter 9, Advanced Continuous Delivery, additionally adds databases to the Continuous Delivery process. The final environment created in this book is presented in the following diagram:

Staging and production environments are equipped with Docker Swarm clusters and therefore multiple instances of the application are run on the cluster. We don't have to think anymore on which exact machine our applications are deployed. All we care about is the number of their instances. The same applies to Jenkins slaves, they are also run on a cluster. The last improvement is the automatic management of the database schemas using Flyway migrations integrated into the delivery process.

I hope you are already excited by what we plan to build throughout this book. We will approach it step by step, explaining every detail and all the possible options in order to help you understand the procedures and tools. After reading this book, you will be able to introduce or improve the Continuous Delivery process in your projects.

Summary

In this chapter, we have introduced the Continuous Delivery process starting from the idea, discussing the prerequisites, to end up with tools that are used in the rest of the book. The key takeaway from this chapter is as follows:

  • The delivery process used currently in most companies has significant shortcomings and can be improved using modern tools for automation
  • The Continuous Delivery approach provides a number of benefits, of which the most significant ones are: fast delivery, fast feedback cycle, and low-risk releases
  • The Continuous Delivery pipeline consists of three stages: Continuous Integration, automated acceptance testing, and configuration management
  • Introducing Continuous Delivery usually requires a change in the organization's culture and structure
  • The most important tools in the context of Continuous Delivery are Docker, Jenkins, and Ansible

In the next chapter, we introduce Docker and present how to build a dockerized application.

Left arrow icon Right arrow icon
Download code icon Download Code

Key benefits

  • Build reliable and secure applications using Docker containers.
  • Create a complete Continuous Delivery pipeline using Docker, Jenkins, and Ansible.
  • Deliver your applications directly on the Docker Swarm cluster.
  • Create more complex solutions using multi-containers and database migrations.

Description

The combination of Docker and Jenkins improves your Continuous Delivery pipeline using fewer resources. It also helps you scale up your builds, automate tasks and speed up Jenkins performance with the benefits of Docker containerization. This book will explain the advantages of combining Jenkins and Docker to improve the continuous integration and delivery process of app development. It will start with setting up a Docker server and configuring Jenkins on it. It will then provide steps to build applications on Docker files and integrate them with Jenkins using continuous delivery processes such as continuous integration, automated acceptance testing, and configuration management. Moving on you will learn how to ensure quick application deployment with Docker containers along with scaling Jenkins using Docker Swarm. Next, you will get to know how to deploy applications using Docker images and testing them with Jenkins. By the end of the book, you will be enhancing the DevOps workflow by integrating the functionalities of Docker and Jenkins.

Who is this book for?

This book is indented to provide a full overview of deep learning. From the beginner in deep learning and artificial intelligence to the data scientist who wants to become familiar with Theano and its supporting libraries, or have an extended understanding of deep neural nets. Some basic skills in Python programming and computer science will help, as well as skills in elementary algebra and calculus.

What you will learn

  • Get to grips with docker fundamentals and how to dockerize an application for the Continuous Delivery process
  • Configure Jenkins and scale it using Docker-based agents
  • Understand the principles and the technical aspects of a successful Continuous Delivery pipeline
  • Create a complete Continuous Delivery process using modern tools: Docker, Jenkins, and Ansible
  • Write acceptance tests using Cucumber and run them in the Docker ecosystem using Jenkins
  • Create multi-container applications using Docker Compose
  • Managing database changes inside the Continuous Delivery process and understand effective frameworks such as Cucumber and Flyweight
  • Build clustering applications with Jenkins using Docker Swarm
  • Publish a built Docker image to a Docker Registry and deploy cycles of Jenkins pipelines using community best practices
Estimated delivery fee Deliver to Turkey

Standard delivery 10 - 13 business days

$12.95

Premium delivery 3 - 6 business days

$34.95
(Includes tracking information)

Product Details

Country selected
Publication date, Length, Edition, Language, ISBN-13
Publication date : Aug 24, 2017
Length: 332 pages
Edition : 1st
Language : English
ISBN-13 : 9781787125230
Vendor :
Docker
Tools :

What do you get with Print?

Product feature icon Instant access to your digital eBook copy whilst your Print order is Shipped
Product feature icon Paperback book shipped to your preferred address
Product feature icon Download this book in EPUB and PDF formats
Product feature icon Access this title in our online reader with advanced features
Product feature icon DRM FREE - Read whenever, wherever and however you want
Estimated delivery fee Deliver to Turkey

Standard delivery 10 - 13 business days

$12.95

Premium delivery 3 - 6 business days

$34.95
(Includes tracking information)

Product Details

Publication date : Aug 24, 2017
Length: 332 pages
Edition : 1st
Language : English
ISBN-13 : 9781787125230
Vendor :
Docker
Tools :

Packt Subscriptions

See our plans and pricing
Modal Close icon
$19.99 billed monthly
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Simple pricing, no contract
$199.99 billed annually
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just $5 each
Feature tick icon Exclusive print discounts
$279.99 billed in 18 months
Feature tick icon Unlimited access to Packt's library of 7,000+ practical books and videos
Feature tick icon Constantly refreshed with 50+ new titles a month
Feature tick icon Exclusive Early access to books as they're written
Feature tick icon Solve problems while you work with advanced search and reference features
Feature tick icon Offline reading on the mobile app
Feature tick icon Choose a DRM-free eBook or Video every month to keep
Feature tick icon PLUS own as many other DRM-free eBooks or Videos as you like for just $5 each
Feature tick icon Exclusive print discounts

Frequently bought together


Stars icon
Total $ 141.97
Learning Continuous Integration with Jenkins
$48.99
Deployment with Docker
$43.99
Continuous Delivery with Docker and Jenkins
$48.99
Total $ 141.97 Stars icon

Table of Contents

9 Chapters
Introducing Continuous Delivery Chevron down icon Chevron up icon
Introducing Docker Chevron down icon Chevron up icon
Configuring Jenkins Chevron down icon Chevron up icon
Continuous Integration Pipeline Chevron down icon Chevron up icon
Automated Acceptance Testing Chevron down icon Chevron up icon
Configuration Management with Ansible Chevron down icon Chevron up icon
Continuous Delivery Pipeline Chevron down icon Chevron up icon
Clustering with Docker Swarm Chevron down icon Chevron up icon
Advanced Continuous Delivery Chevron down icon Chevron up icon

Customer reviews

Rating distribution
Full star icon Full star icon Full star icon Full star icon Empty star icon 4
(3 Ratings)
5 star 33.3%
4 star 33.3%
3 star 33.3%
2 star 0%
1 star 0%
lowtek Mar 26, 2019
Full star icon Full star icon Full star icon Empty star icon Empty star icon 3
Some good tips but didn't really learn anything new. If you already understand CI and CD and use Jenkins/Docker often this won't be anything new for you either. Some good best practices listed but again nothing groundbreaking
Amazon Verified review Amazon
Joginder Raperia Jan 23, 2018
Full star icon Full star icon Full star icon Full star icon Empty star icon 4
Explains Jenkins, Ansible and Docker in a very easy way. Gives a very good starting point for anyone new in the area of Containers, Configuration Management and Continuous Delivery.
Amazon Verified review Amazon
Adam Nov 02, 2017
Full star icon Full star icon Full star icon Full star icon Full star icon 5
This is the best book available on the topic, period. I've also read a couple of other books on the topic, but they are outdated.This is the most up to date manual at the moment on how to implement Continuous Delivery with Docker and Jenkins.It covers Jenkins 2 and uses the latest features of Docker (swarm, stack).A big thank you to the author who put together this great book.Thank you.
Amazon Verified review Amazon
Get free access to Packt library with over 7500+ books and video courses for 7 days!
Start Free Trial

FAQs

What is the delivery time and cost of print book? Chevron down icon Chevron up icon

Shipping Details

USA:

'

Economy: Delivery to most addresses in the US within 10-15 business days

Premium: Trackable Delivery to most addresses in the US within 3-8 business days

UK:

Economy: Delivery to most addresses in the U.K. within 7-9 business days.
Shipments are not trackable

Premium: Trackable delivery to most addresses in the U.K. within 3-4 business days!
Add one extra business day for deliveries to Northern Ireland and Scottish Highlands and islands

EU:

Premium: Trackable delivery to most EU destinations within 4-9 business days.

Australia:

Economy: Can deliver to P. O. Boxes and private residences.
Trackable service with delivery to addresses in Australia only.
Delivery time ranges from 7-9 business days for VIC and 8-10 business days for Interstate metro
Delivery time is up to 15 business days for remote areas of WA, NT & QLD.

Premium: Delivery to addresses in Australia only
Trackable delivery to most P. O. Boxes and private residences in Australia within 4-5 days based on the distance to a destination following dispatch.

India:

Premium: Delivery to most Indian addresses within 5-6 business days

Rest of the World:

Premium: Countries in the American continent: Trackable delivery to most countries within 4-7 business days

Asia:

Premium: Delivery to most Asian addresses within 5-9 business days

Disclaimer:
All orders received before 5 PM U.K time would start printing from the next business day. So the estimated delivery times start from the next day as well. Orders received after 5 PM U.K time (in our internal systems) on a business day or anytime on the weekend will begin printing the second to next business day. For example, an order placed at 11 AM today will begin printing tomorrow, whereas an order placed at 9 PM tonight will begin printing the day after tomorrow.


Unfortunately, due to several restrictions, we are unable to ship to the following countries:

  1. Afghanistan
  2. American Samoa
  3. Belarus
  4. Brunei Darussalam
  5. Central African Republic
  6. The Democratic Republic of Congo
  7. Eritrea
  8. Guinea-bissau
  9. Iran
  10. Lebanon
  11. Libiya Arab Jamahriya
  12. Somalia
  13. Sudan
  14. Russian Federation
  15. Syrian Arab Republic
  16. Ukraine
  17. Venezuela
What is custom duty/charge? Chevron down icon Chevron up icon

Customs duty are charges levied on goods when they cross international borders. It is a tax that is imposed on imported goods. These duties are charged by special authorities and bodies created by local governments and are meant to protect local industries, economies, and businesses.

Do I have to pay customs charges for the print book order? Chevron down icon Chevron up icon

The orders shipped to the countries that are listed under EU27 will not bear custom charges. They are paid by Packt as part of the order.

List of EU27 countries: www.gov.uk/eu-eea:

A custom duty or localized taxes may be applicable on the shipment and would be charged by the recipient country outside of the EU27 which should be paid by the customer and these duties are not included in the shipping charges been charged on the order.

How do I know my custom duty charges? Chevron down icon Chevron up icon

The amount of duty payable varies greatly depending on the imported goods, the country of origin and several other factors like the total invoice amount or dimensions like weight, and other such criteria applicable in your country.

For example:

  • If you live in Mexico, and the declared value of your ordered items is over $ 50, for you to receive a package, you will have to pay additional import tax of 19% which will be $ 9.50 to the courier service.
  • Whereas if you live in Turkey, and the declared value of your ordered items is over € 22, for you to receive a package, you will have to pay additional import tax of 18% which will be € 3.96 to the courier service.
How can I cancel my order? Chevron down icon Chevron up icon

Cancellation Policy for Published Printed Books:

You can cancel any order within 1 hour of placing the order. Simply contact customercare@packt.com with your order details or payment transaction id. If your order has already started the shipment process, we will do our best to stop it. However, if it is already on the way to you then when you receive it, you can contact us at customercare@packt.com using the returns and refund process.

Please understand that Packt Publishing cannot provide refunds or cancel any order except for the cases described in our Return Policy (i.e. Packt Publishing agrees to replace your printed book because it arrives damaged or material defect in book), Packt Publishing will not accept returns.

What is your returns and refunds policy? Chevron down icon Chevron up icon

Return Policy:

We want you to be happy with your purchase from Packtpub.com. We will not hassle you with returning print books to us. If the print book you receive from us is incorrect, damaged, doesn't work or is unacceptably late, please contact Customer Relations Team on customercare@packt.com with the order number and issue details as explained below:

  1. If you ordered (eBook, Video or Print Book) incorrectly or accidentally, please contact Customer Relations Team on customercare@packt.com within one hour of placing the order and we will replace/refund you the item cost.
  2. Sadly, if your eBook or Video file is faulty or a fault occurs during the eBook or Video being made available to you, i.e. during download then you should contact Customer Relations Team within 14 days of purchase on customercare@packt.com who will be able to resolve this issue for you.
  3. You will have a choice of replacement or refund of the problem items.(damaged, defective or incorrect)
  4. Once Customer Care Team confirms that you will be refunded, you should receive the refund within 10 to 12 working days.
  5. If you are only requesting a refund of one book from a multiple order, then we will refund you the appropriate single item.
  6. Where the items were shipped under a free shipping offer, there will be no shipping costs to refund.

On the off chance your printed book arrives damaged, with book material defect, contact our Customer Relation Team on customercare@packt.com within 14 days of receipt of the book with appropriate evidence of damage and we will work with you to secure a replacement copy, if necessary. Please note that each printed book you order from us is individually made by Packt's professional book-printing partner which is on a print-on-demand basis.

What tax is charged? Chevron down icon Chevron up icon

Currently, no tax is charged on the purchase of any print book (subject to change based on the laws and regulations). A localized VAT fee is charged only to our European and UK customers on eBooks, Video and subscriptions that they buy. GST is charged to Indian customers for eBooks and video purchases.

What payment methods can I use? Chevron down icon Chevron up icon

You can pay with the following card types:

  1. Visa Debit
  2. Visa Credit
  3. MasterCard
  4. PayPal
What is the delivery time and cost of print books? Chevron down icon Chevron up icon

Shipping Details

USA:

'

Economy: Delivery to most addresses in the US within 10-15 business days

Premium: Trackable Delivery to most addresses in the US within 3-8 business days

UK:

Economy: Delivery to most addresses in the U.K. within 7-9 business days.
Shipments are not trackable

Premium: Trackable delivery to most addresses in the U.K. within 3-4 business days!
Add one extra business day for deliveries to Northern Ireland and Scottish Highlands and islands

EU:

Premium: Trackable delivery to most EU destinations within 4-9 business days.

Australia:

Economy: Can deliver to P. O. Boxes and private residences.
Trackable service with delivery to addresses in Australia only.
Delivery time ranges from 7-9 business days for VIC and 8-10 business days for Interstate metro
Delivery time is up to 15 business days for remote areas of WA, NT & QLD.

Premium: Delivery to addresses in Australia only
Trackable delivery to most P. O. Boxes and private residences in Australia within 4-5 days based on the distance to a destination following dispatch.

India:

Premium: Delivery to most Indian addresses within 5-6 business days

Rest of the World:

Premium: Countries in the American continent: Trackable delivery to most countries within 4-7 business days

Asia:

Premium: Delivery to most Asian addresses within 5-9 business days

Disclaimer:
All orders received before 5 PM U.K time would start printing from the next business day. So the estimated delivery times start from the next day as well. Orders received after 5 PM U.K time (in our internal systems) on a business day or anytime on the weekend will begin printing the second to next business day. For example, an order placed at 11 AM today will begin printing tomorrow, whereas an order placed at 9 PM tonight will begin printing the day after tomorrow.


Unfortunately, due to several restrictions, we are unable to ship to the following countries:

  1. Afghanistan
  2. American Samoa
  3. Belarus
  4. Brunei Darussalam
  5. Central African Republic
  6. The Democratic Republic of Congo
  7. Eritrea
  8. Guinea-bissau
  9. Iran
  10. Lebanon
  11. Libiya Arab Jamahriya
  12. Somalia
  13. Sudan
  14. Russian Federation
  15. Syrian Arab Republic
  16. Ukraine
  17. Venezuela