For this chapter, the only technical requirements include having access to a laptop and an account for one of the cloud providers mentioned in the CI/CD providers section (preferably Microsoft’s Azure Pipelines – don’t worry, it’s free).

Once you have reviewed how pipelines are created, you’ll be able to apply the same concepts to other cloud providers and their pipeline strategies.

In this section, we’ll learn about what continuous integration and continuous deployment mean to developers.

Continuous Integration (CI) is the process of merging all developers’ code into a mainline to trigger an automatic build process so that you can quickly identify issues with a code base using unit tests and code analysis.

When a developer checks their code into a branch, it’s reviewed by peer developers. Once accepted, it’s merged into a mainline and automatically starts a build process. This build process will be covered shortly.

Continuous Deployment (CD) is the process of consistently creating software to deploy it at any time.

Once everything has been built through the automated process, the build prepares the compiled code and creates artifacts. These artifacts are used for consistent deployments across various environments, such as development, staging, and production.

The benefits of implementing a CI/CD pipeline outweigh not having one:

• Automated Testing: When a commit is triggered, your tests are automatically executed along with your build. Think of this as someone always checking your code on commit.
• Faster Feedback Loops: As a developer, it’s always great to receive immediate feedback to find out whether something works or not. If you receive an email where the build broke, you’re on your own.
• Consistent Builds: Once you have a project being built on a build server, you can create builds on-demand – and consistently – with tests.
• Collaboration Between Teams: We’re all in this together and CI/CD includes developers, system administrators, project managers/SCRUM masters, and QA testers, to name a few, to accomplish the goal of creating great software.

In this section, we reviewed the definition of what continuous integration and continuous deployment mean when developing software in an automated fashion and the benefits of implementing a CI/CD pipeline.

In the next section, we’ll learn about certain code practices to avoid when automating software builds.

In this section, we’ll cover certain aspects of your code and how they could impact the deployment of your software. Such software issues could include code not compiling (broken builds), avoiding relative path names, and making sure you wrote proper unit tests. These are a couple of the common errors I’ve experienced over the years; in this section, I’ll also provide solutions on how to fix them.

Before we review a CI pipeline, there are a few caveats we should address beforehand. Even though we covered a lot in the previous chapter regarding version control, your code needs to be in a certain state to achieve “one-button” builds.

In the following sections, you’ll learn how to prepare your code so that it’s “CI/CD-ready” and examine the problems you could experience when deploying your software and how to avoid them.

Building Flawlessly

If a new person is hired and starts immediately, you want them to hit the ground running and begin developing software without delay. This means being able to point them to a repository and pull the code so that you can immediately run the code with minimal setup.

I say “minimal setup” because there may be permissions involved to gain access to certain resources in the company so that they can be run locally.

Nevertheless, the code should be in a runnable state, send you to a simple screen of some kind, and notify the user to follow up on a permissions issue or provide some notification to resolve the problem.

In the previous chapter, we mentioned how the code should compile at all times. This means the following:

• The code should always compile after a clone or checkout
• Unit tests should be included with the build, not in separate projects
• Your commit messages to version control should be meaningful (they may be used for Release Notes)

These standards allow your pipeline to fall into the pit of success. They help you create a build even faster and easier when your code is in a clean state.

Avoiding Relative Path Names with File-based Operations

One of the troublesome issues I’ve seen over the years when it comes to web applications is how files are accessed in a web application.

I’ve also seen file-based operations through a web page, where files were moved using relative paths and it went wrong. It involved deleting directories and it didn’t end well.

For example, let’s say you had a relative path to an image, as follows:

../images/myimage.jpg

Now, let’s say you’re sitting on a web page, such as https://localhost/kitchen/chairs.

If you went back one directory, you’d be in the kitchen with a missing image, not at the root of the website. According to your relative path, you’re looking for an image directory at https://localhost/kitchen/images/myimage.jpg.

To make matters worse, if you’re using custom routing, this may not even be the normal path, and who knows where it’s looking for the image.

The best approach when preparing your code is to use a single slash (/) at the beginning of your URL since it’s considered “absolute:”

/images/myimage.jpg

This makes it easier to navigate to the root when you’re locating files on a website, regardless of what environment you’re in. It doesn’t matter if you are on https://www.myfakewebsite.com/ or http://localhost/, the root is the root, and you’ll always find your files when using a single slash at the beginning of your sources.

Confirming that your Unit Tests are Unit Tests

Tests in your code are created to provide checks and balances so that your code works as expected. Each test needs to be examined carefully to confirm it isn’t doing anything out of the ordinary.

Unit tests are considered tests against code in memory, whereas integration tests are tests that require ANY external resources:

• Do your tests access any files? Integration test.
• Do you connect to a database to test something? Integration test.
• Are you testing business logic? Unit test.

As you’re beginning to surmise, when you build your application on another machine, cloud services do not have access to your database server and also may not have the additional files you need for each test to pass.

If you are accessing external resources, it may be a better approach to refactor your tests into something a little more memory-driven. I’ll explain why in Chapter 7, when we’ll cover unit testing.

Creating Environment Settings

Whether you are in the middle of a project or are clicking Create New Project… for the first time, you need a way to create environment settings for your web application.

In ASP.NET Core applications, we are given appsettings.json and appsettings.Development.json configuration files out of the box. The appsettings.json file is meant to be a base configuration file and, depending on the environment, each appsettings file is applied and only existing properties are overwritten to the appsettings.json file.

One common example of this is connection strings and application paths. Depending on the environment, each file will have its own settings.

The environments need to be defined upfront as well. There will always be a development and release environment. There may be an option to create another environment called QA on another machine somewhere, so an appsettings.qa.json file would be required with its own environment-specific settings.

Confirm that these settings have been saved for each relevant environment since they are important in a CI/CD pipeline. These environment settings should always be checked into version control with your solution/project to assist the pipeline in deploying the right settings to the right environment.

In this section, we covered ways to prepare your code for a CI/CD pipeline by making sure we can build immediately after cloning or pulling the repository down locally, why we should avoid relative-based file paths, and confirmed we were using environment-specific application settings, making it easy to build and deploy our application.

With your code checked in, we can now move forward and describe all of the stages of a common pipeline.

In this section, we’ll cover the steps of what a common pipeline includes for building software when using a CI/CD service. When you reach the end of this section, you’ll understand every step of the process in a common pipeline so that you can produce quality software.

A CI pipeline is a collection of steps required to code, build, test, and deploy software. Each step is not owned by a particular person but by a team working together and focusing on the goal to produce exceptional software. The good news is that if you followed the previous chapter’s recommendations, you’re already ahead of the game.

Each company’s pipeline can vary from product to product, but there will always be a common set of steps for a CI process. It depends on how detailed your pipeline becomes based on your needs. The stages in the pipelines can be influenced by each stakeholder involved in the process. Of course, pulling code and building and testing are required for the developers, but a QA team requires the finished product (artifact) to be sent to another server for test purposes.

Figure 2.1 shows one common pipeline:

Figure 2.1 – One example of a build pipeline

As shown in Figure 2.1, the process is sequential when creating a software deployment. Here’s a summary of the steps:

1. Pull code from a single repository.
2. Build the application.
3. Run unit tests/code analysis against the code that was built in step 2.
4. Create the artifacts.
5. Create a container (optional).
6. Deploy the artifact(s) to a server (development/QA/staging/production).

Now that we’ve defined a common pipeline, let’s dig deeper into each step to learn what each process includes when you’re building your software.

In the following subsections, we’ll examine each process in detail based on the steps defined here.

Pulling Code

Before we build the application, we need to identify the project we’re building in our pipeline. The pipeline service requires a repository location. Once you’ve provided the repository URL, the service can prepare the repository for compilation on their server.

In the previous section, we mentioned why your code needs to compile flawlessly after cloning. The code is cloned and built on a completely different machine from yours. If the application only works on your computer and no one else’s, as the saying goes, “We’ll have to ship your computer to all of our users.” While this is a humorous saying in the industry, it’s generally frowned upon when writing and deploying software in the real world.

Each of the DevOps services has its benefits. For example, Azure Pipelines can examine your repository and make assumptions based on the structure of your project.

After analyzing the project, it uses a file format called YAML (pronounced Ya-mel) to define how the project should be built. While YAML is now considered a standard in the industry, we won’t deep-dive into everything YAML encompasses. YAML functionality could be a book on its own.

Azure takes your project’s assumptions and creates a YAML template on how it should build your application.

It knows how to compile the application, identify whether a container is included in the project, and also retrieve NuGet packages before performing the build.

One last thing to mention is that most DevOp services allow one repository per project. The benefits of this approach include the following:

• Simplicity: It’s simpler to manage and build one application as opposed to orchestrating hundreds of applications in a project.
• Collaboration: Instead of multiple teams focusing on one large project, it’s easier to have one or two smaller teams working on a single, more manageable project.
• Faster builds: CI/CD pipelines are meant to provide fast feedback and even faster improvement. The smaller the project, the faster a build, test, and deployment will occur.

With that said, we are now ready to build the application.

Building the application

As mentioned previously, YAML files define how the service proceeds with building your application.

It’s always a good practice to confirm the YAML file contains everything you need before building. If you have a simple project, the boilerplate included in the wizard may be all you need, but it allows you to make updates in case additional files are required, or other application checks.

It may take a couple of attempts to massage the YAML file, but once you get the file in a stable state, it’s great to see everything work as expected.

Make sure you have retrieved all your code before building the application. If this step fails, the process kicks out of the pipeline.

If you checked in bad code and the build fails, the proper authorities (developers or administrators) will be notified based on the alert level and you’ll be given the dunce hat or the stuffed monkey for breaking the build until someone else breaks it.

Next, we’ll focus on running unit tests and other tests against the application.

Running Unit Tests/Code Analysis

Once the build is done, we can move forward with the unit tests and/or code analysis.

Unit tests should run against the compiled application. This includes unit tests and integration tests, but as we mentioned previously, be wary of integration tests. The pipeline services may not have access to certain resources, causing your tests to fail.

Unit tests, by nature, should be extremely fast. Why? Because you don’t want to wait for 30 minutes for unit tests to run (which is painful). If you have unit tests taking that long, identify the longest-running unit tests and refactor them.

Once the code has been compiled and loaded, unit tests should be running every 10-30 seconds as a general guideline since they are memory-based.

While unit and integration tests are common in most testing scenarios, there are additional checks you can add to your pipeline, which include identifying security issues and code metrics to generate reports at the end of your build.

Next, our build creates artifacts to be used for deployments.

Creating Artifacts

Once the build succeeds and all of the tests pass, the next step is to create an artifact of our build and store it in a central location.

As a general rule, it’s best to only create your binaries once. Once they’ve been built, they’re available at a moment’s notice. These artifacts can deploy a version to a server on a whim without going through the entire build process again.

The artifacts should be tamper-proof and never be modified by anyone. If there is an issue with the artifact, the pipeline should start from the beginning and create a new artifact.

Let’s move on to containers.

Creating a Container

Once you have created the self-contained artifact, an optional step is to build a container around it or install the artifact in the container. While most enterprises use various platforms and environments, such as Linux or Windows, “containerizing” an application with a tool such as Docker allows it to run on any platform while isolating the application.

With containers considered a standard in the industry, it makes sense to create a container so that it can easily be deployed to any platform, such as Azure, Amazon Web Services (AWS), or Google Cloud Provider. Again, this is an optional step, but it’s becoming an inevitable one in the industry.

When creating a new project with Visual Studio, you automatically get a container wrapper through a generated Docker file. This Dockerfile defines how the container will allow access to your application.

Once you’ve added the Dockerfile to your project, Azure identifies this as a container project and creates the container with the included project.

Lastly, we’ll examine deploying the software.

Deploying the software

Once everything has been generated, all we need to do is deploy the software.

Remember the environment settings in your appsettings.json file? This is where they come in handy for deployments.

Based on your environment, you can assign a task to merge the appropriate environment JSON file into the appsettings.json file on deployment.

Once you have your environment settings in order, you can define the destinations of your deployments any way you like.

Deployments can range from FTP-ing or WebDeploy-ing the artifact or pushing the container to a server somewhere. All of these options are available out of the box.

However, you must deploy the same way to every environment. The only thing that changes is the appsettings file.

After a successful (or unsuccessful) deployment, a report or notification should be sent to everyone involved in the deployment’s outcome.

In this section, we learned what a common pipeline includes and how each step relies on a successful previous step. If one step fails throughout the pipeline, the process immediately stops. This “conveyor belt” approach to software development provides repeatable steps, quality-driven software, and deployable software.

In this section, we’ll learn about two ways to recover from a failed software deployment. After finishing this section, you’ll know how to use these two approaches to make a justified decision on recovering from a bad deployment.

In a standard pipeline, companies sometimes experience software glitches when deploying to a web server. Users may see an error message when they perform an action on the website.

What do you do when the software doesn’t work as expected? How does this work in the DevOps pipeline?

Every time you build software, there’s always a chance something could go wrong. You always need a backup plan before the software is deployed.

Let’s cover the two types of recovery methods we can use when software deployments don’t succeed.

Falling Backward (or fallback)

If various bugs were introduced into the product and the previous version doesn’t appear to have these errors, it makes sense to revert the software or fall back to the previous version.

In a pipeline, the process at the end creates artifacts, which are self-contained, deployable versions of your product.

Here is an example of falling backward:

1. Your software deployment was a success last week and was marked as version 1.1 (v1.1).
2. Over 2 weeks, development created two new features for the software and wanted to release them as soon as possible.
3. A new build was created and released called version 1.3 (v1.3).
4. While users were using the latest version (v1.3), they experienced issues with one of the new features, causing the website to show errors.
5. Since the previous version (v1.1) doesn’t have this issue and the impact is not severe, developers can redeploy v1.1 to the server so that users can continue to be productive again.

This type of release is called falling backward.

If you have to replace a current version (v1.3) with a previous version (v1.1) (except for databases, which I’ll cover in a bit), you can easily identify and deploy the last-known artifact.

Falling Forward

If the fallback approach isn’t a viable recovery strategy, the alternative is to fall forward.

When falling forward, the product team accepts the deployment with errors (warts and all) and continues to move forward with newer releases while placing a high priority on these errors and acknowledging the errors will be fixed in the next or future release.

Here is a similar example of falling forward:

1. Again, a software deployment was successful last week and was marked as version 1.5 (v1.5).
2. Over another 2 weeks, development created another new large feature for the software.
3. A new build was created and released called version 1.6 (v1.6).
4. While users were using the latest version (v1.6), they experienced issues with one of the new features, causing the website to show errors.
5. After analysis, the developers realized this was a “quick fix,” created the proper unit tests to show it was fixed, pushed a new release through the pipeline, and immediately deployed the fixed code in a new release (v1.7).

This type of release is called falling forward.

The product team may have to examine each error and make a decision as to which recovery method is the best approach for the product’s reputation.

For example, if product features such as business logic or user interface updates are the issue, the best recovery method may be to fall forward since the impact on the system is minimal and a user’s workflow is not interrupted and productive.

However, if code and database updates are involved, the better approach would be to fall back – that is, restore the database and use a previous version of the artifact.

If it’s a critical feature and reverting is not an option, a “hotfix” approach (as mentioned in the previous chapter) may be required to patch the software.

Again, it depends on the impact each issue has left on the system as to which recovery strategy is the best approach.

In this section, we learned about two ways to recover from unsuccessful software deployments: falling backward and falling forward. While neither option is a mandatory choice, each approach should be weighed heavily based on the error type, the recovery time of the fix, and the software’s deployment schedule.

Deploying application code is one thing but deploying databases can be a daunting task if not done properly. There are two pain points when deploying databases: structure and records.

With a database’s structure, you have the issue of adding, updating, and removing columns/fields from tables, along with updating the corresponding stored procedures, views, and other table-related functions to reflect the table updates.

With records, the process isn’t as tricky as changing a table’s structure. The frequency of updating records is not as regular, but when it does, happen that’s when you either want to seed a database with default records or update those seed records with new values.

The following sections will cover some common practices when deploying databases in a CI/CD pipeline.

Backing up Before Deploying

Since company data is essential to a business, it’s mandatory to back it up before making any modifications or updates to the database.

One recommendation is to make the entire database deploy a two-step process: back up the database, then apply the database updates.

The DevOps team can include a pre-deployment script to automatically back up the database before applying the database updates. If the backup was successful, you can continue deploying your changes to the database. If not, you can immediately stop the deployment and determine the cause of failure.

As discussed in the previous section, this is necessary for a “fallback” approach instead of a “fall forward” strategy.

Creating a Strategy for Table Structures

One strategy for updating a table is to take a non-destructive approach:

• Adding a column: When adding columns, place a default value on the column for when a record is created. This will prevent the application from erroring out when you add a record, notifying the user that a field didn’t have a value or is required.
• Updating/renaming a column: Updating a column is a little different because you may be changing a data type or value in the database. If you’re changing the column name and/or type to something else, add a new column with the new column type, make sure you default the value, and proceed to use it in your application code. Once the code is solid and is performing as expected, remove the old column from the table and then from your code.
• Removing a column: There are several different ways to handle this process. If the field was created with a default value, make the appropriate changes in your application code to stop using the column. When records are added to the table, the default value won’t create an error. Once the application code has been updated, rename the column in the table instead of deleting it. If your code is still using it, you’ll be able to identify the code issue and fix it. Once your code is running without error, it’ll be safe to remove the column from your table.

While making the appropriate changes to table structures, don’t forget about updating the additional database code to reflect the table changes, including stored procedures, views, and functions.

Creating a Database Project

If your Visual Studio solution connects to a database, there’s another project type you need to add to your solution called the Database Project type. When you add this project to your solution, it takes a snapshot of your database and adds it to your project as code.

Why include this in your solution? There are three reasons to include it in your solution:

1. It provides a database schema as T-SQL when you create a database from scratch.
2. It allows you to version your database, in keeping with the Infrastructure as Code (IaC) paradigm.
3. When you’re building your solution in Visual Studio, it automatically generates a DAC file from your Database Project for deployment with the option to attach a custom script. With the DAC included in your solution, the pipeline can deploy and update the database with the DAC file first. Once the database deployment (and backup) is finished, the pipeline can deploy the artifact.

As you can see, it’s pretty handy to include with your solution.

Using Entity Framework Core’s Migrations

Entity Framework has come a long way since its early days. Migrations are another way to include database changes through C# as opposed to T-SQL.

Upon creating a migration, Entity Framework Core takes a snapshot of the database and DbContext and creates the delta between the database schema and DbContext using C#.

With the initial migration, the entire C# code is generated with an Up() method.

Any subsequent migrations will contain an Up() method and a Down() method for upgrading and downgrading the database, respectively. This allows developers to save their database delta changes, along with their code changes.

Entity Framework Core’s migrations are an alternative to using DACs and custom scripts. These migrations can perform database changes based on the C# code.

If you require seed records, then you can use Entity Framework Core’s .HasData() method for easily creating seed records for tables.

In this section, we learned how to prepare our database deployment by always creating a backup, looked at a common strategy for adding, updating, and deleting table fields, and learned how to deploy databases in a CI/CD pipeline using either a DAC or Entity Framework Core’s migrations.

Now that we’ve learned how a standard pipeline works, in this section, we’ll look at the different types of pipeline providers.

The three types of providers are on-premises, off-premises, and hybrid.

On-premises (meaning on-site or on-premises) relates to the software you own, which you can use to build your product at your company’s location. An advantage of on-premises build services is that once you purchase the software, you own it; there isn’t a subscription fee. So, if there’s a problem with the build server, you can easily look at the software locally to identify and fix the problem.

Off-premises (or cloud) providers are the more common services used nowadays. Since everyone wants everything yesterday, it’s quicker to set up and is usually an immediate way to create a software pipeline.

As you can guess, hybrid services are a mix of on-premises and off-premises services. Some companies like to keep control of certain aspects of software development and send the artifacts to a remote server for deployment purposes.

While hybrid services are an option, it makes more sense to use off-premises services for automated software builds.

In this section, we learned about three types of providers: on-premises, off-premises, and hybrid services. While these services are used in various companies, the majority of companies lean toward off-premises (or cloud) services to automate their software builds.