There are no technical requirements for this chapter.

If you were to list all of the different definitions and descriptions of DevOps, there would be many. However, as different as these might be, they will most likely share several concepts. These are collaboration, continuous delivery of business value, and breaking down silos.

With all of the technical discussion in the rest in this book, it is important not to overlook the value proposition for adopting DevOps, namely, that it will help you to improve the way that you continuously deliver value to your end users. To do this, you have to decrease the time between starting work on a new feature and the first user using it in production. This means that you not only have to write the software but also deliver and operate it.

Over the last decade, the way we write software has fundamentally changed. More and more companies are now adopting an Agile way of working to increase the efficiency of their software development. More and more teams are now working in short iterations or sprints to create new increments of a product in quick succession. However, creating potentially shippable increments faster and faster does not create any value in itself. Only when each new version of your software is also released to production and used by your end users does it start delivering value.

In traditional organizations, developers and operators are often located in different departments and taking software into production includes a hand-off, often with a formal ceremony around it. In such an organization, it can be hard to accelerate that delivery to production along with the speed at which development can create new versions.

Next to that, development and operations departments often have conflicting goals. While a development department is rewarded for creating many changes as fast as possible, operation departments are rewarded for limiting downtime and preventing issues. The latter is often best achieved by having as few changes as possible. The conflict here is clear—both departments have optimizations for one subgoal, as shown in the following diagram:

This defeats the purpose of these subgoals, which comes from the shared, overarching goal of quickly taking in new versions while maintaining stability. Precisely this conflict between developmental and operational goals is one of the things that should disappear in a DevOps culture. In such a culture, developers and operations teams should work together on delivering new versions to production in a fast and reliable manner and share responsibility for both subgoals.

While it is good to know that DevOps is a cultural movement, tools and automation are an important part of that culture. In this book, the focus will be on these tools and how to use them to implement many of the practices that come with a DevOps culture. In other words, this book will be mostly on the products and processes associated with DevOps. If you want to learn more about the cultural side of things, about the people, there are many other books to read.

The rest of this section will explore the relation between DevOps to see how they complement each other. The focus will be on Agile techniques and prices for work management. We will also discuss the goals and benefits of a DevOps culture.

If you take a look at Agile, you might notice that part of it is the focus on business value and shortening the time between the delivery of a new business value. From that perspective, adopting DevOps is a logical next step after Agile. Agile advocates that the software development teams' responsibilities should extend forward by engaging with users and other stakeholders to more quickly deliver valuable potentially shippable products. DevOps is all about not just creating something that might be shipped, but really shipping it as well. With Agile and DevOps combined, you can create an end-to-end, continuous flow of value to your users.

One of the things you need to be able to do this is a common approach to managing the work to be done for everyone involved. In the next section, you will find some pointers on how to incorporate operational concerns in the way you manage your work.

When you are starting to increase the collaboration between development and operations, you will quickly notice that they have to cope with different types of work. In development, a large part of the work is planned: user stories and bugs that are picked up from a backlog. On the other hand, for operations, a large part of their work is unplanned. They respond to warnings and alerts from systems and requests or tickets from users or developers.

Integrating these two, especially if developers and operators are located on the same team, can be challenging. To see how you can deal with this, let's explore the following approach:

1. First, switch to a flow-based way of working for developers.
2. Next, allow for operations to also list their work in the same work management system as developers using synchronizations. You can also choose to implement fastlaning, an approach to expedite urgent work.
3. Finally, you might choose to decommission existing ticketing tools for operations if possible.

Fastlaning is an approach to organizing work that allows for both planned and unplanned work by visualizing two separate lanes of work. To do this, the Scrum board is extended with a Kanban-like board on the top. This is the fast lane. On the Kanban board, urgent but unplanned work is added. Any work added to this lane is picked up by the team with the highest priority. Only when there is no work remaining in the fast lane is work from the Scrum board with planned work picked up. Whenever new work is added to the fast lane, this takes priority again. Often, there is the agreement though that work in progress is finished before switching to work in the fast lane.

The first thing to consider is transitioning the way developers work from batch-wise to flow-based. An example of a batch-wise way of working is Scrum. If you are using the Scrum framework, you are used to picking up a batch of work every two to four weeks and focus on completing all of that work within that time window. Only when that batch is done do you deliver a potentially shippable product.

When changing to a flow-based approach, you try to focus not on a batch, but on just one thing only. You work on that one work item and drive it completely until it's done before you start on the next. This way, there is no longer a sprint backlog, only a product backlog. The advantage of this approach is that you no longer decide which work to perform upfront, but whenever you are free to start on new work, you can pick up the next item from the backlog. In an environment where priorities quickly shift, this allows you to react to change more quickly.

These changes to the way developers organize their work make it easier to include operations in work management, but there is also another benefit. When developers are focusing on getting a single work item done instead of a whole sprint at once, you can also increase the number of times you can deliver a small portion of value to your users.

After the development team changes the way it organizes its work, it should now be easier for developers to also list their planned work on the shared backlog and pull work from that backlog when they have time to work on it. They now also have a place where they can list their unplanned work.

However, there might still be an existing ticketing system where requests for operations are dropped by users or automatically created by monitoring tools. While Azure DevOps has a great API to rework this integration to directly create work items in Azure DevOps, you might first choose to create a synchronization between your existing ticketing tool and Azure Boards. There are many integration options available and there is a lot of ongoing work in this area. This way, operators can slowly move from their old tool to the new one, since they are now in sync. Of course, the goal is for them to move over to the same tool as the developers completely.

With the work of developers and operators in the same work management tool, you will notice that you have a mix of planned and unplanned, often urgent, work in the system. To ensure that urgent work gets the attention and precedence it deserves, you can introduce what is called a fastlane to your sprint board. In the following screenshot, you can see an example of an Azure Board that is set up for fastlaning production issues:

The use of this horizontal split in the board is to only work on tasks in the regular lane when there is no work to be picked up in the fast lane.

After creating a shared work management system between development and operations, there is much opportunity to increase the amount of collaboration between them. When this collaboration is taking off, old ticketing systems that were used by operations might now slowly be decommissioned over time. Integrations from monitoring tools can be shifted to the new shared tools and the number of tickets between developers and operators should slowly decrease as they find new ways of working together.

At this point, you might be wondering about the point of it all. What are the benefits of DevOps and what is there in it for you, your colleague, and your organization? The most common goal of adopting DevOps is to achieve a reduction in cycle time. Cycle time is the time between starting work on a new feature and the moment that the first user can use it. The way this is achieved, by automation, also serves the goals of lower change failure rate, lower Mean Time To Repair (MTTR) and lower planned downtime.

Next to all that, there might also be other benefits such as increased employee satisfaction, less burnout and stress, and better employee retention. This is attributed to the removal of opposing goals between developers and operators.

For a while, there was doubt whether DevOps really works, and whether these goals were really met, and whether the extra benefits were really achieved, as this was only shown using case studies. The downside of this is that case studies are often only available for successful cases and not for unsuccessful cases. This all changed in 2018 when the book Accelerate came out. This book shows, based on years of quantitative research, that modern development practices such as DevOps really contribute to reaching IT goals and organizational goals.

To measure where you currently stand as a team or organization and the impact of DevOps on you, there are several metrics that you could start recording. As always when working with metrics or Key Performance Indicators (KPIs), make sure that you do not encourage people to game the system by looking only at the numbers. Several interesting metrics are detailed in the following sections and if you go over them, you will notice that they are all about encouraging flow.

Cycle time and lead time are metrics that come from Lean and Kanban and are used to measure the time needed to realize a change. Cycle time is the amount of time between starting work on a feature and users being able to use that feature in production. The lower cycle time, the quicker you can react to changing requirements or insights. Lead time is the amount of time between requesting a feature and realizing that feature. It is the time between adding work to the backlog and the start of implementation.

When you add cycle time and lead time together, you are calculating another metric, the time to market. This last one is often an important business metric when developing software. Minimizing both cycle time and lead time will hence have a business impact.

Another thing you can measure is the amount of work in progress at any point in time. DevOps is focusing on the flow of value to the user. This implies that everyone should, if possible, be doing only one thing at a time and completely finish that before moving on to something else. This reduces the amount of time spent on task switching and the amount of time spent on not yet complete work. Measuring how many things a team works on in parallel and reporting on this can encourage this.

You can even go as far as putting actual limits on the amount of work that can be in progress. The following is a small part of the earlier screenshot, showing that these work-in-progress limits can even be shown in the tool:

The goal is to have as little work in progress at the same time as possible.

A third metric is the mean time to recovery. How long does it take you to restore service in case of a (partial) outage? In the past, companies focused on reducing the mean time between failures. This used to be the mean indicator of the stability of a product. However, this metric encourages limiting the number of changes going to production. The unwanted consequence often is that outages, though maybe rare, last long and are hard to fix.

Measuring the mean time to recovery shifts the attention to how quickly you can remediate an outage. If you can fix outages quickly, you achieve the same, namely, minimizing the amount of downtime without sacrificing the rate of change. The goal is to minimize the time to recovery.

Finally, you can measure the number of changes delivered to production and the percentage of that which is not successful. Increasing the rate of change implies that you are more often delivering value to your users, hence realizing a flow of value. Also, by measuring not just the number of failures, but the percentage that fails, you are actually encouraging many small, successful changes instead of encouraging limiting the number of changes overall.

Your goals should be to increase the rate of change while lowering the change failure rate.

At this point, you might be wondering, how do I change my organization to foster this culture and reap all of these benefits? The next section will answer this for you.

Well, maybe your organizational structure does not have to change at all. DevOps has to start with a cultural change: openness, empathy, and collaboration are values that need to be encouraged. But still, changing your organizational structure might help to accelerate this.

Traditionally, developers and operators are often organized in disparate teams or even different departments—organized in teams with people that have a similar skill set and responsibility. A common change to organizations is changing this structure, by pivoting and organizing teams behind a common goal, a single product, or a group of features, for example.

Now you will need teams with different skill sets and responsibilities, teams most likely with developers and operators. It is important to realize that forcing such a change upon these people might not be the best way forward. Often, it works best to start with changing the culture, encouraging cooperation, and then this organizational change might come about in a natural way.

Finally, it is important to recognize one anti-pattern at this point. Some companies are trying to implement DevOps by hiring specialized DevOps engineers and positioning them between development and operations, interacting with both. While this, at first, might seem like a good idea, this goes against the DevOps values. If you do this, you are not breaking silos down, but you are adding a third one. You are not decreasing the number of hand-offs, you are most likely increasing them. Also, collaboration between developers and operations is often not enhanced by separating them using another organizational structure and you might not see any increase in value to your end users at all.

Now that you know what DevOps is and you have a clear understanding of how you can form a DevOps team, it is time to explore how to start achieving your goals.

Since you are not the first team going on this journey, you can learn from the experiences of those before you. One example is the Microsoft team that built Azure DevOps. Being in the rare position that they can use their own product for developing their product, they have learned a great deal about what makes DevOps successful. From this, they have identified seven key DevOps practices and seven DevOps habits that many successful DevOps teams share:

Now it is important to realize that just copying the motions described will not guarantee success. Just as with Agile, you will have to spend time to really understand these practices and habits, where they come from, and what they contribute to a continuous flow of value to your end users.

The following sections explore all of these practices and habits in more detail. Keep these in the back of your mind while reading the rest of this book. While the rest of this book will mostly focus on technical means of how to do things, do not forget that these are only means. The real value comes from mindset and creating a culture that is focused on creating a continuous flow of value to your customers.

This section discusses all seven DevOps practices in turn. As you will quickly see, they are highly related and it is quite hard to practice one without the other. For example, test automation is highly related to continuous integration and continuous deployment.

Configuration management is about versioning the configuration of your application and the components it relies on, along with your application itself. Configuration is kept in source control and takes the form of, for example, JSON or YAML files that describe the desired configuration of your application. These files are the input for tools such as Ansible, Puppet, or PowerShell DSC that configure your environment and application. These tools are often invoked from a continuous deployment pipeline.

The desired state can also be reapplied at an interval, even if there are no changes made to the intended configuration. This way, it is ensured that the actual configuration stays correct and that manual changes are automatically revoked. We call this the prevention of configuration drift. Configuration drift occurs over time due to servers being added or removed over time, or manual, ad hoc interventions by administrators. Of course, this implies that intended updates to the configuration are done in source control and only applied using tools.

Configuration management or configuration as code is highly related to infrastructure as code. The two are often intertwined and on some platforms, the difference between the two might even feel artificial. Configuration as code will be discussed in detail in Chapter 6, Infrastructure and Configuration as Code.

Release management is about being in control of which version of your software is deployed to which environment. Versions are often created using continuous integration and delivery pipelines. These versions, along with all of the configuration needed, are then stored as immutable artifacts in a repository. From here on, release management tools are used to plan and control how these versions are deployed to one or more environments. Example controls are manual approvals and automated queries of open work and quality checks before allowing deployment to a new environment.

Release management is related to continuous deployment and focuses more on controlling the flow of versions through the continuous deployment pipeline. Chapter 6, Infrastructure and Configuration as Code, will cover configuration as code as part of release management.

Continuous integration is a practice where every developer integrates their own work with that of the other developers in the team at least once a day and preferably more often. This means that every developer should push their work to the repository at least once a day and a continuous integration build verifies that their work compiles and that all unit tests run. It is important to understand that this verification should not run only on the code that the developer is working on in isolation. The real value comes when the work is also integrated with the work of others.

When integrating changes often and fast, problems with merging changes are less frequent and if they occur, are often less difficult to solve. In Chapter 2, Everything Starts with Source Control, you will learn more about how to set up your source control repositories to make this possible. In Chapter 3, Moving to Continuous Integration, you will learn about setting up a continuous integration build.

Continuous deployment is the practice of automatically deploying every new version of sufficient quality to production. When practicing continuous deployment, you have a fully automated pipeline that takes in every new version of your application (every commit), results in a new release, and starts deploying it to one or more environments. The first environment is often called test and the final environment will be production.

In this pipeline, there are multiple steps that verify the quality of the software, before letting it proceed to the next environment. If the quality is not sufficient, the release is aborted and will not propagate to the next environment. The premise behind this approach is that, in the pipeline, you try to prove that you cannot take the current version to the next environment. If you fail to prove so, you assume it is ready for further progression.

Only when a release has gone through all environments in the pipeline, it is deployed to production. Whenever a release cannot progress to the next environment, that release will be completely canceled. While you might be inclined to fix the reason for the failure and then restart deployment from the point where it failed, it is important not to do so. The changes you made at that point are after all not validated by all of the controls that the version has already passed through. The only way to validate the new version as a whole is by starting the pipeline from the start. You can see this clearly in the following diagram:

In Chapter 4, Continuous Deployment, you will learn about setting up continuous deployment using Azure DevOps Pipelines.

When writing an application, the binaries that you are building have to be running somewhere, on some application host. An example of such an application host can be a web server such as IIS or Apache. Next to an application host, we might need a database and some messaging solution. All of this together we call the infrastructure for our application. When practicing infrastructure as code, you are keeping a description of this infrastructure in your source code repository, alongside your application code.

When the time comes to release a new version of the application and you require one or more changes in the infrastructure, you are executing this description of your desired infrastructure using tools such as Chef, Puppet, PowerShell DSC, or Azure ARM templates. The execution of such a description is idempotent, which means that it can be executed more than once and the end result is the same. This is because your description of the infrastructure describes the desired state you want the infrastructure to be in and not a series of steps to be executed. Those steps to be executed, if there are any, are automatically determined by your tool of choice. Applying the desired state can also be done automatically in a continuous deployment pipeline and is often executed before updating the application code.

The big advantage of this is that you can now easily create a new environment, where the infrastructure is guaranteed to be the same as in your other environments. Also, the problem of configuration drift, where the infrastructure between your different environment slowly diverges, is no longer possible since every time, you apply the desired state again to every environment and they are forced.

Chapter 6, Infrastructure and Configuration as Code, of this book will discuss infrastructure as code in more detail.

To continuously deliver value to your end users, you will have to release fast and often. This has implications for the way you test your application. You can no longer execute manual tests when you release your application every few minutes. This means that you have to automate as many of your tests as possible.

You will most likely want to create multiple test suites for your applications that you run at different stages of your delivery pipeline. Fast unit tests that run within a few minutes and that are executed whenever a new pull request is opened should give your team very quick feedback on the quality of their work and should catch most of the errors. Next, the team should run one or more slower test suites later in the pipeline to further increase your confidence in the quality of a version of your application.

All of this should limit the amount of manual testing to a bare minimum and allow you to automatically deploy new versions of your application with confidence.

Chapter 8, Continuous Testing, of this book will cover test automation in detail.

This last practice is about learning all about how your application is doing in production. Gathering metrics such as response times and the number of requests will tell you about how the systems are performing. Capturing errors is also part of performance monitoring and allows you to start fixing problems without having to wait on your customers to contact you about them.

In addition to that, you can gather information on which parts of the application are more or less frequently used and whether new features are being picked up by users. Learning about usage patterns provides you with great insights into how customers really use your applications and common scenarios they are going through.

Chapter 9, Security and Compliance, and Chapter 10, Application Monitoring, will go into detail on learning about both your application and your users' behavior in production.

The seven habits of successful DevOps teams are more concerned with culture and your attitude while developing and delivering software and less with technical means than DevOps practices are. Still, it is important to know and understand these habits since they will help to make DevOps adoption easier.

You will notice that developing these habits will reinforce the use of the practices enumerated previously and the tools you use to implement them. And of course, this holds the other way around as well.

An important part of working Agile is creating teams that are largely self-directed and can make decisions without (too many) dependencies outside the team. Such a team will hence often include multiple roles, including a product owner that owns one or more features and is empowered to decide on the way forward with those.

However, this autonomy also comes with the responsibility to align the work of the team with the direction the whole product is taking. It is important to develop ways of aligning the work of tens or hundreds of teams with each other, in such a way that everyone can sail their own course, but the fleet as a whole stays together as well.

The best-case scenario is that teams take it upon themselves to align to the larger vision, instead of taking directions every now and then.

Another habit is that of rigorous management of technical debt. The term debt in itself suggests that there is a cost (interest) associated with the delay of addressing an issue. To keep moving at a constant pace and not slowly lose speed over time, it is crucial to keep the number of bugs or architectural issues to a minimum and only tolerate so much. Within some teams this is even formalized in agreements. For example, a team can agree that the number of unfixed bugs should never exceed the number of team members. This means, that if a team has four members and a ninth bug is reported that no new work will be undertaken until at least one bug should be fixed.

It is important to accept that users receive no value from code that has been written until they are actually using it. Focusing on the flow of value to a user means that code has to be written, tested, and delivered and should be running in production before you are done. Focusing on this habit can really drive cooperation between disciplines and teams.

In many modern development methodologies, there is a product owner who is responsible for ordering all of the work in the backlog, based on the business value. This owner, as the expert, is responsible for maximizing the value delivered by the development team by ordering all items based on business value (divided by effort).

However, recent research has shown that, even though the product owner is an expert, they cannot correctly predict which features will bring the most value to users. Roughly one third of the work from a team actually adds value for users, and even worse while, another third actually decreases value. For this reason, you can switch your backlog from features or user stories to the hypothesis you want to prove or disprove. You create only a minimal implementation or even just a hint of a feature in the product and then measure whether it is picked up by users. Only when this happens do you expand the implementation of the feature.

Performance measurements should be taken in your production environment, not (just) in an artificial load test environment. There is nothing wrong with executing load tests before going to production if they deliver value to you. However, the real performance is done in the production environment. And it should be measured there and compared with previous measurements.

This holds also for usage statistics, patterns, and many, many other performance indicators. They can all be automatically gathered using production metrics.

A live-site culture promotes the idea that anything that happens in the production environment takes precedence over anything else. Next, anything that threatens production, is about to go to production, or hinders going to production at any time gets priority. Only when these are all in order is the attention shifted to future work.

Also, a part of a live-site culture is ensuring that anything that disturbed the operation of the service is thoroughly analyzed—not to find out who to blame or fire but to find out how to prevent this from happening again. Prevention is preferably done by shifting left, for example, detecting an indicator of a repeat incident earlier in the pipeline.

Finally, a successful DevOps team treats its servers and infrastructure as cattle, not as pets. This means that infrastructure is spun up when needed and disregarded as soon as it is not needed anymore. The ability to do this is fueled by configuration and infrastructure as code. This might even go so far as creating a new production environment for every new deployment and just deleting the old production environment after switching all traffic from the old environment to the new one.

Besides keeping these DevOps practices and habits in mind, there are certain stages that you will go through while trying to move to a DevOps culture in your organization. The next section will take you through it.

When you are trying to move to a DevOps culture in your organization, this is going to take time. There are motions you have to go through while everyone in your organization embraces the changes they have to make to their individual way of working. Others that have gone before you have gone through the following five steps or stages that might help you. Knowing about them can help you to accelerate your own journey. These steps were first published in the 2018 State of DevOps Report and are discussed in the following sections.

A common first step on the road to a DevOps culture is the adoption of Agile. At a minimum, there are good tools for source control, and often a company standard and continuous integration and delivery are being rolled out. Teams are also working together to normalize the stack they develop software for. For example, one or two cloud vendors are chosen and other deployment platforms are phased out. The same goes for tools for other purposes—they are standardized where possible. Homebrewed solutions are replaced with industry standards.

In this stage, teams work on further reducing the variation between and within applications and the development and operations teams that work on them, working together on aligning operating systems, libraries, and tools. Also, in this stage, deployment processes are changed to reduce the amount of variation between them and configuration and infrastructure are often moved to source control.

Remaining issues between development and operations are cleaned up, ensuring that outputs of the development team are precisely what the operations team expects. Also, collaboration starts to grow between the two and they are able to work together without external dependencies on creating and delivering changes.

In this stage, the infrastructure that is used by developers and operations becomes fully aligned. Everything is deployed from source control and the same scripts are being used by both teams.

Before DevOps, virtual machines or hosting environments were often requested from operations, by developers manually or through ticketing systems. Provisioning was done manually by operators, which could take days or sometimes even weeks.

Self-service capabilities means that environments are no longer created manually, but through self-service API's that operations teams make available to developers.

This way, developers are fully able to create and destroy environments on their own. They can create and test changes on their own and send them off or schedule them for automated deployment.

In this chapter, you learned what DevOps is (and what it is not) and its relation to Agile. Moving to a DevOps culture helps you to break down conflicting targets for developers on one side and operators on the other side. This to empower them to work together on continuously delivering value to your end users, organizing their work in a single backlog and working off a single board, while respecting the differences in their ways of working. Organizing developers and operators in product-oriented teams is the next important step in creating like-minded, goal-oriented teams.

Moving to DevOps can bring many benefits and you now know how these can be measured to continuously keep improving. Next, you learned about the DevOps habits and practices that many successful DevOps team exhibit. Mastering these yourself and with your team will enable you to go through your own DevOps evaluation. All this is with the aim to continuously deliver value to your users.

The next chapter will discuss the topic of source control and how to organize your application sources to enable DevOps flows.

As we conclude, here is a list of questions for you to test your knowledge regarding this chapter's material. You will find the answers in the Assessments section of the Appendix:

1. True or false: Development and operations departments often have conflicting goals.
2. True or false: The seven DevOps practices discussed in this chapter are unrelated and one can be easily practiced without the other.
3. Which of the following is not a part of the five stages of DevOps evolution?
   1. Normalizing the technology stack
   2. Automating infrastructure delivery
   3. Standardizing and reducing variability
   4. Hiring a group of DevOps engineers to automate the delivery of applications
4. What is fastlaning?
5. Describe in your own words, in a few lines, what the essence of DevOps is.

There are many other resources that you might find helpful to learn more about DevOps culture and the DevOps way of thinking. Some of them are as follows:

• The Phoenix Project, by Gene Kim, Kevin Behr, and George Spafford
• Effective DevOps, by Jennifer Davis and Katherine Daniels
• Accelerate, by Nicole Forsgren, Jez Humble and Gene Kim
• Interview with Sam Guckenheimer, available at https://devops.com/11626/
• Microsoft Case Study on their DevOps Journey, available at http://stories.visualstudio.com/devops/
• The 2018 State of DevOps Report, available at http://info.puppet.com/Eficode-Puppet-State-of-DevOps-Report.html
• More information on assessing existing development processes can be found at https://docs.microsoft.com/en-us/learn/modules/assess-your-development-process/index
• More information about different Agile approaches and how to support them using Azure Boards can be found at https://docs.microsoft.com/en-us/learn/modules/choose-an-agile-approach/index