In this article by Mitesh Soni, the author of the book DevOps for Web Development, provides some insight into DevOps movement, benefits of DevOps culture, Lifecycle of DevOps, how Jenkins 2.0 is bridging the gaps between Continuous Integration and Continuous Delivery using new features and UI improvements, installation and configuration of Jenkins 2.0.

(For more resources related to this topic, see here.)

Understanding the DevOps movement

Let's try to understand what DevOps is. Is it a real, technical word? No, because DevOps is not just about technical stuff. It is also neither simply a technology nor an innovation. In simple terms, DevOps is a blend of complex terminologies. It can be considered as a concept, culture, development and operational philosophy, or a movement.

To understand DevOps, let's revisit the old days of any IT organization. Consider there are multiple environments where an application is deployed. The following sequence of events takes place when any new feature is implemented or bug fixed:

1. The development team writes code to implement a new feature or fix a bug. This new code is deployed to the development environment and generally tested by the development team.
2. The new code is deployed to the QA environment, where it is verified by the testing team.
3. The code is then provided to the operations team for deploying it to the production environment.
4. The operations team is responsible for managing and maintaining the code.

Let's list the possible issues in this approach:

• The transition of the current application build from the development environment to the production environment takes weeks or months.
• The priorities of the development team, QA team, and IT operations team are different in an organization and effective, and efficient co-ordination becomes a necessity for smooth operations.
• The development team is focused on the latest development release, while the operations team cares about the stability of the production environment.
• The development and operations teams are not aware of each other's work and work culture.
• Both teams work in different types of environments; there is a possibility that the development team has resource constraints and they therefore use a different kind of configuration. It may work on the localhost or in the dev environment.
• The operations team works on production resources and there will therefore be a huge gap in the configuration and deployment environments. It may not work where it needs to run—the production environment.
• Assumptions are key in such a scenario, and it is improbable that both teams will work under the same set of assumptions.
• There is manual work involved in setting up the runtime environment and configuration and deployment activities. The biggest issue with the manual application-deployment process is its nonrepeatability and error-prone nature.
• The development team has the executable files, configuration files, database scripts, and deployment documentation. They provide it to the operations team. All these artifacts are verified on the development environment and not in production or staging.
• Each team may take a different approach for setting up the runtime environment and the configuration and deployment activities, considering resource constraints and resource availability.
• In addition, the deployment process needs to be documented for future usage. Now, maintaining the documentation is a time-consuming task that requires collaboration between different stakeholders.
• Both teams work separately and hence there can be a situation where both use different automation techniques.
• Both teams are unaware of the challenges faced by each other and hence may not be able to visualize or understand an ideal scenario in which the application works.
• While the operations team is busy in deployment activities, the development team may get another request for a feature implementation or bug fix; in such a case, if the operations team faces any issues in deployment, they may try to consult the development team, who are already occupied with the new implementation request. This results in communication gaps, and the required collaboration may not happen.
• There is hardly any collaboration between the development team and the operations team. Poor collaboration causes many issues in the application's deployment to different environments, resulting in back-and-forth communication through e-mail, chat, calls, meetings, and so on, and it often ends in quick fixes.
• Challenges for the development team:
  • The competitive market creates pressure of on-time delivery.
  • They have to take care of production-ready code management and new feature implementation.
  • The release cycle is often long and hence the development team has to make assumptions before the application deployment finally takes place. In such a scenario, it takes more time to fix the issues that occurred during deployment in the staging or production environment.
• Challenges for the operations team:
  • Resource contention: It's difficult to handle increasing resource demands
  • Redesigning or tweaking: This is needed to run the application in the production environment
  • Diagnosing and rectifying: They are supposed to diagnose and rectify issues after application deployment in isolation

The benefits of DevOps

This diagram covers all the benefits of DevOps:

Collaboration among different stakeholders brings many business and technical benefits that help organizations achieve their business goals.

The DevOps lifecycle – it's all about "continuous"

Continuous Integration(CI),Continuous Testing(CT), and Continuous Delivery(CD) are significant part of DevOps culture. CI includes automating builds, unit tests, and packaging processes while CD is concerned with the application delivery pipeline across different environments. CI and CD accelerate the application development process through automation across different phases, such as build, test, and code analysis, and enable users achieve end-to-end automation in the application delivery lifecycle:

Continuous integration and continuous delivery or deployment are well supported by cloud provisioning and configuration management. Continuous monitoring helps identify issues or bottlenecks in the end-to-end pipeline and helps make the pipeline effective.

Continuous feedback is an integral part of this pipeline, which directs the stakeholders whether are close to the required outcome or going in the different direction.

"Continuous effort – not strength or intelligence – is the key to unlocking our potential"
                                                                                            -Winston Churchill

Continuous integration

What is continuous integration? In simple words, CI is a software engineering practice where each check-in made by a developer is verified by either of the following:

• Pull mechanism: Executing an automated build at a scheduled time
• Push mechanism: Executing an automated build when changes are saved in the repository

This step is followed by executing a unit test against the latest changes available in the source code repository.

The main benefit of continuous integration is quick feedback based on the result of build execution. If it is successful, all is well; else, assign responsibility to the developer whose commit has broken the build, notify all stakeholders, and fix the issue.

Read more about CI at http://martinfowler.com/articles/continuousIntegration.html.

So why is CI needed? Because it makes things simple and helps us identify bugs or errors in the code at a very early stage of development, when it is relatively easy to fix them. Just imagine if the same scenario takes place after a long duration and there are too many dependencies and complexities we need to manage. In the early stages, it is far easier to cure and fix issues; consider health issues as an analogy, and things will be clearer in this context.

Continuous integration is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

CI is a significant part and in fact a base for the release-management strategy of any organization that wants to develop a DevOps culture.

Following are immediate benefits of CI:

• Automated integration with pull or push mechanism
• Repeatable process without any manual intervention
• Automated test case execution
• Coding standard verification
• Execution of scripts based on requirement
• Quick feedback: build status notification to stakeholders via e-mail
• Teams focused on their work and not in the managing processes

Jenkins, Apache Continuum, Buildbot, GitLabCI, and so on are some examples of open source CI tools. AnthillPro, Atlassian Bamboo, TeamCity, Team Foundation Server, and so on are some examples of commercial CI tools.

Continuous integration tools – Jenkins

Jenkins was originally an open source continuous integration software written in Java under the MIT License. However, Jenkins 2 an open source automation server that focuses on any automation, including continuous integration and continuous delivery.

Jenkins can be used across different platforms, such as Windows, Ubuntu/Debian, Red Hat/Fedora, Mac OS X, openSUSE, and FreeBSD. Jenkins enables user to utilize continuous integration services for software development in an agile environment. It can be used to build freestyle software projects based on Apache Ant and Maven 2/Maven 3. It can also execute Windows batch commands and shell scripts.

It can be easily customized with the use of plugins. There are different kinds of plugins available for customizing Jenkins based on specific needs for setting up continuous integration. Categories of plugins include source code management (the Git, CVS, and Bazaar plugins), build triggers (the Accelerated Build Now and Build Flow plugins), build reports (the Code Scanner and Disk Usage plugins), authentication and user management (the Active Directory and GitHub OAuth plugins), and cluster management and distributed build (Amazon EC2 and Azure Slave plugins).

To know more about all plugins, visit https://wiki.jenkins-ci.org/display/JENKINS/Plugins.

To explore how to create a new plugin, visit https://wiki.jenkins-ci.org/display/JENKINS/Plugin+tutorial.

To download different versions of plugins, visit https://updates.jenkins-ci.org/download/plugins/.

Visit the Jenkins website at http://jenkins.io/.

Jenkins accelerates the software development process through automation:

Key features and benefits

Here are some striking benefits of Jenkins:

• Easy install, upgrade, and configuration.
• Supported platforms: Windows, Ubuntu/Debian, Red Hat/Fedora/CentOS, Mac OS X, openSUSE, FreeBSD, OpenBSD, Solaris, and Gentoo.
• Manages and controls development lifecycle processes.
• Non-Java projects supported by Jenkins: Such as .NET, Ruby, PHP, Drupal, Perl, C++, Node.js, Python, Android, and Scala.
• A development methodology of daily integrations verified by automated builds.
• Every commit can trigger a build.
• Jenkins is a fully featured technology platform that enables users to implement CI and CD.
• The use of Jenkins is not limited to CI and CD. It is possible to include a model and orchestrate the entire pipeline with the use of Jenkins as it supports shell and Windows batch command execution. Jenkins 2.0 supports a delivery pipeline that uses a Domain-Specific Language (DSL) for modeling entire deployments or delivery pipelines.
• Pipeline as code provides a common language—DSL—to help the development and operations teams to collaborate in an effective manner.
• Jenkins 2 brings a new GUI with stage view to observe the progress across the delivery pipeline.
• Jenkins 2.0 is fully backward compatible with the Jenkins 1.x series.
• Jenkins 2 now requires Servlet 3.1 to run.
• You can use embedded Winstone-Jetty or a container that supports Servlet 3.1 (such as Tomcat 8).
• GitHub, Collabnet, SVN, TFS code repositories, and so on are supported by Jenkins for collaborative development.
• Continuous integration: Automate build and test—automated testing (continuous testing), package, and static code analysis.
• Supports common test frameworks such as HP ALM Tools, Junit, Selenium, and MSTest.
• For continuous testing, Jenkins has plugins for both; Jenkins slaves can execute test suites on different platforms.
• Jenkins supports static code analysis tools such as code verification by CheckStyle and FindBug. It also integrates with Sonar.
• Continuous delivery and continuous deployment: It automates the application deployment pipeline, integrates with popular configuration management tools, and automates environment provisioning.
• To achieve continuous delivery and deployment, Jenkins supports automatic deployment; it provides a plugin for direct integration with IBM uDeploy.
• Highly configurable: Plugins-based architecture that provides support to many technologies, repositories, build tools, and test tools; it has an open source CI server and provides over 400 plugins to achieve extensibility.
• Supports distributed builds: Jenkins supports "master/slave" mode, where the workload of building projects is delegated to multiple slave nodes.
• It has a machine-consumable remote access API to retrieve information from Jenkins for programmatic consumption, to trigger a new build, and so on.
• It delivers a better application faster by automating the application development lifecycle, allowing faster delivery.

The Jenkins build pipeline (quality gate system) provides a build pipeline view of upstream and downstream connected jobs, as a chain of jobs, each one subjecting the build to quality-assurance steps. It has the ability to define manual triggers for jobs that require intervention prior to execution, such as an approval process outside of Jenkins. In the following diagram Quality Gates and Orchestration of Build Pipeline is illustrated:

Jenkins can be used with the following tools in different categories as shown here:

Installing Jenkins

Jenkins provides us with multiple ways to install it for all types of users. We can install it on at least the following operating systems:

• Ubuntu/Debian
• Windows
• Mac OS X
• OpenBSD
• FreeBSD
• openSUSE
• Gentoo
• CentOS/Fedora/Red Hat

One of the easiest options I recommend is to use a WAR file. A WAR file can be used with or without a container or web application server. Having Java is a must before we try to use a WAR file for Jenkins, which can be done as follows:

1. Download the jenkins.war file from https://jenkins.io/.

2. Open command prompt in Windows or a terminal in Linux, go to the directory where the jenkins.war file is stored, and execute the following command:

   java – jar jenkins.war

3. Once Jenkins is fully up and running, as shown in the following screenshot, explore it in the web browser by visiting http://localhost:8080.

   

4. By default, Jenkins works on port 8080. Execute the following command from the command line:

   java -jar jenkins.war --httpPort=9999

5. For HTTPS, use the following command:

   java -jar jenkins.war --httpsPort=8888

6. Once Jenkins is running, visit the Jenkins home directory. In our case, we have installed Jenkins 2 on a CentOS 6.7 virtual machine.

7. Go to /home/<username>/.jenkins, as shown in the following screenshot. If you can't see the .jenkins directory, make sure hidden files are visible. In CentOS, press Ctrl+H to make hidden files visible.

   

Setting up Jenkins

Now that we have installed Jenkins, let's verify whether Jenkins is running. Open a browser and navigate to http://localhost:8080 or http://<IP_ADDRESS>:8080. If you've used Jenkins earlier and recently downloaded the Jenkins 2 WAR file, it will ask for a security setup.

To unlock Jenkins, follow these steps:

1. Go to the .Jenkins directory and open the initialAdminPassword file from the secrets subdirectory:

   

2. Copy the password in that file, paste it in the Administrator password box, and click on Continue, as shown here:

   

3. Clicking on Continue will redirect you to the Customize Jenkins page. Click on Install suggested plugins:

   

4. The installation of the plugins will start. Make sure that you have a working Internet connection.

   

5. Once all the required plugins have been installed, you will seethe Create First Admin User page. Provide the required details, and click on Save and Finish:

   

6. Jenkins is ready! Our Jenkins setup is complete. Click on Start using Jenkins:

   

Get Jenkins plugins from https://wiki.jenkins-ci.org/display/JENKINS/Plugins.

Summary

We have covered some brief details on DevOps culture and Jenkins 2.0 and its new features. DevOps for Web Developmentprovides more details on extending Continuous Integration to Continuous Delivery and Continuous Deployment using Configuration management tools such as Chef and Cloud Computing platforms such Microsoft Azure (App Services) and AWS (Amazon EC2 and AWS Elastic Beanstalk), you refer at https://www.packtpub.com/networking-and-servers/devops-web-development. To get more details Jenkins, refer to JenkinsEssentials, https://www.packtpub.com/application-development/jenkins-essentials.

Resources for Article:

Further resources on this subject:

• Setting Up and Cleaning Up [article]
• Maven and Jenkins Plugin [article]
• Exploring Jenkins [article]