Before we dive deeper into this chapter, we need to establish an understanding of some of the basic terms and concepts around software engineering. For starters, how do we define software engineering and in what ways does it differ from software development and programming in general? To begin answering this question, we will start by examining the formal definition of software engineering, as published in IEEE's Standard Glossary of Software Engineering Terminology ^[7]:

The main takeaway from this definition is that authoring code is just one of the many facets of software engineering. At the end of the day, any capable programmer can take a well-defined specification and convert it into a fully functioning program without thinking twice about the need to produce clean and maintainable code. A disciplined software engineer, on the other hand, would follow a more systematic approach by applying common design patterns to ensure that the produced piece software is extensible, easier to test, and well documented in case another engineer or engineering team assumes ownership of it in the future.

Besides the obvious requirement for authoring high-quality code, the software engineer is also responsible for thinking about other aspects of the systems that will be built. Some questions that the software engineer must be able to answer include the following:

• What are the business use cases that the software needs to support?
• What components comprise the system and how do they interact with each other?
• Which technologies will be used to implement the various system components?
• How will the software be tested to ensure that its behavior matches the customer's expectations?
• How does load affect the system's performance and what is the plan for scaling the system?

To be able to answer these questions, the software engineer needs a special set of skills that, as you are probably aware, go beyond programming. These extra responsibilities and required skills are the main factors that differentiate a software engineer from a software developer.

As we discussed in the previous section, software engineering is an inherently complex, multi-stage process. In an attempt to manage this complexity, organizations around the world have invested a lot of time and effort over the years to break the process down into a set of well-defined stages and train their engineering staff to efficiently deal with each stage.

Some software engineers strive to work across all the stages of the Software Development Life Cycle (SDLC), while others have opted to specialize in and master a particular stage of the SDLC. This gave rise to a variety of software engineering roles, each one with a different set of responsibilities and a required set of skills. Let's take a brief look at the most common software engineering roles that you may encounter when working with both small- and large-sized organizations.

The software engineer (SWE) is the most common role that you are bound to interact with in any organization, regardless of its size. Software engineers play a pivotal role not only in designing and building new pieces of software, but also in operating and maintaining existing and legacy systems.

Depending on their experience level and technical expertise, SWEs are classified into three categories:

• Junior engineer: A junior engineer is someone who has recently started their software development career and lacks the necessary experience to build and deploy production-grade software. Companies are usually keen on hiring junior engineers as it allows them to keep their hiring costs low. Furthermore, companies often pair promising junior engineers with senior engineers in an attempt to grow them into mid-level engineers and retain them for longer.
• Mid-level engineer: A typical mid-level engineer is someone who has at least three years of software development experience. Mid-level engineers are expected to have a solid grasp of the various aspects of the software development life cycle and are the ones who can exert a significant impact on the amount of code that's produced for a particular project. To this end, they not only contribute code, but also review and offer feedback to the code that's contributed by other team members.
• Senior engineer: This class of engineer is well-versed in a wide array of disparate technologies; their breadth of knowledge makes them ideal for assembling and managing software engineering teams, as well as serving as mentors and coaches for less senior engineers. From their years of experience, senior engineers acquire a deep understanding of a particular business domain. This trait allows them to serve as a liaison between their teams and the other, technical or non-technical, business stakeholders.

Another way to classify software engineers is by examining the main focus of their work:

• Frontend engineers work exclusively on software that customers interact with. Examples of frontend work include the UI for a desktop application, a single-page web application for a software as a service (SaaS) offering, and a mobile application running on a phone or other smart device.
• Backend engineers specialize in building the parts of a system that implement the actual business logic and deal with data modeling, validation, storage, and retrieval.
• Full stack engineers are developers who have a good understanding of both frontend and backend technologies and no particular preference of doing frontend or backend work. This class of developers is more versatile as they can easily move between teams, depending on the project requirements.

The software development engineer in test (SDET) is a role whose origins can be traced back to Microsoft's engineering teams. In a nutshell, SDETs are individuals who, just like their SWE counterparts, take part in software development, but their primary focus lies in software testing and performance.

An SDET's primary responsibility is to ensure that the development team produces high-quality software that is free from defects. A prerequisite for achieving this goal is to be cognizant of the different types of approaches to testing software, including, but not limited to, unit testing, integration testing, white/black-box testing, end-to-end/acceptance testing, and chaos testing. We will be discussing all of these testing approaches in more detail in the following chapters.

The main tool that SDETs use to meet their goals is testing automation. Development teams can iterate much faster when a Continuous Integration (CI) pipeline is in place to automatically test their changes across different devices and CPU architectures. Besides setting up the infrastructure for the CI pipeline and integrating it with the source code repository system that the team uses, SDETs are often tasked with authoring and maintaining a separate set of tests. These tests fall into the following two categories:

• Acceptance tests: A set of scripted end-to-end tests to ensure that the complete system adheres to all the customer's business requirements before a new version is given the green light for a release.
• Performance regression tests: Another set of quality control tests that monitor a series of performance metrics across builds and alert you when a metric exceeds a particular threshold. These types of tests prove to be a great asset in the case where a service-level agreement (SLA) has been signed that makes seemingly innocuous changes to the code (for example, switching to a different data structure implementation) that may trigger a breach of the SLA, even though all the unit tests pass.

Finally, SDETs collaborate with support teams to transform incoming support tickets into bug reports that the development team can work on. The combination of software development and debugging skills, in conjunction with the SDET's familiarity with the system under development, makes them uniquely capable of tracking down the location of bugs in production code and coming up with example cases (for example, a particular data input or a sequence of actions) that allow developers to reproduce the exact set of conditions that trigger each bug.

The role of the site reliability engineer (SRE) came into the spotlight in 2016 when Google published a book on the subject of Site Reliability Engineering ^[4]. This book outlined the best practices and strategies that are used internally by Google to run their production systems and has since led to the wide adoption of the role by the majority of companies operating in the SaaS space.

The term was initially coined sometime around 2003 by Ben Treynor, the founder of Google's site reliability team. A site reliability engineer is a software engineer with a strong technical background who also focuses on the operations side of deploying and running production-grade services.

According to the original role definition, SREs spend approximately 50% of their time developing software and the other 50% dealing with ops-related aspects such as the following:

• Working on support tickets or responding to alerts
• Being on-call
• Running manual tasks (for example, upgrading systems or running disaster recovery scenarios)

It is in the best interests of SREs to increase the stability and reliability of the services they operate. After all, no one enjoys being paged at 2 a.m. when a service melts down due to a sudden spike in the volume of incoming requests. The end goal is always to produce services that are highly available and self-healing; services that can automatically recover from a variety of faults without the need for human intervention.

The basic mantra of SREs is to eliminate potential sources of human errors by automating repeated tasks. One example of this philosophy is the use of a Continuous Deployment (CD) pipeline to minimize the amount of time that's required to deploy software changes to production. The benefits of this type of automation become apparent when a critical issue affecting production is identified and a fix must be deployed as soon as possible.

Ultimately, software is designed and built by humans so bugs will undoubtedly creep in. Rather than relying on a rigorous verification process to prevent defects from being deployed to production, SREs operate under the premise that we live in a non-perfect world: systems do crash and buggy software will, at some point, get deployed to production. To detect defective software deployments and mitigate their effects on end users, SREs set up monitoring systems that keep track of various health-related metrics for each deployed service and can trigger automatic rollbacks if a deployment causes an increase in a service's error rate.

In a world where complex, monolithic systems are broken down into multiple microservices and continuous delivery has become the new norm, debugging older software releases that are still deployed out in the wild becomes a major pain point for software engineers.

To understand why this can be a pain point, let's take a look at a small example: you arrive at work on a sunny Monday morning only to find out that one of your major customers has filed a bug against the microservice-based software your team is responsible for. To make things even worse, that particular customer is running a long-term support (LTS) release of the software, which means that some, if not all, of the microservices that the run on the customer's machines are based on code that is at least a couple of hundred commits behind the current state of development. So, how can you actually come up with a bug reproducer and check whether the bug has already been fixed upstream?

This is where the concept of reproducible builds comes into play. By reproducible builds, we mean that at any point in time we should be able to compile a particular version of all the system components where the resulting artifacts match, bit by bit, the ones that have been deployed by the customer.

A release engineer (RE) is effectively a software engineer who collaborates with all the engineering teams to define and document all the required steps and processes for building and releasing code to production. A prerequisite for a release engineer is having deep knowledge of all the tools and processes that are required for compiling, versioning, testing, and packaging software. Typical tasks for REs include the following:

• Authoring makefiles
• Implementing workflows for containerizing software artifacts (for example, as Docker or .rkt images)
• Ensuring all teams use exactly the same build tool (compilers, linkers, and so on) versions and flags
• Ensuring that builds are both repeatable and hermetic: changes to external dependencies (for example, third-party libraries) between builds of the same software version should have no effect on the artifacts that are produced by each build

The last role that we will be discussing in this section, and one that you will only probably encounter when working on bigger projects or collaborating with large organizations, is the system architect. While software engineering teams focus on building the various components of the system, the architect is the one person who sees the big picture: what components comprise the system, how each component must be implemented, and how all the components fit and interact with each other.

In smaller companies, the role of the architect is usually fulfilled by one of the senior engineers. In larger companies, the architect is a distinct role that's filled by someone with both a solid technical background and strong analytical and communication skills.

Apart from coming up with a high-level, component-based design for the system, the architect is also responsible for making decisions regarding the technologies that will be used during development and setting the standards that all the development teams must adhere to.

Even though architects have a technical background, they rarely get to write any code. As a matter of fact, architects tend to spend a big chunk of their time in meetings with the various internal or external stakeholders, authoring design documents or providing technical direction to the software engineering teams.

The software engineering definition from the previous section alludes to the fact that software engineering is a complicated, multi-stage process. In an attempt to provide a formal description of these stages, academia has put forward the concept of the SDLC.

Over the years, there has been an abundance of alternative model proposals for facilitating software development. The following diagram is a timeline illustrating the years when some of the most popular SDLC models were introduced:

In the upcoming sections, we will explore each of the preceding models in more detail.

The waterfall model is probably the most widely known model out there for implementing the SDLC. It was introduced by Winston Royce in 1970 ^[11] and defines a series of steps that must be sequentially completed in a particular order. Each stage produces a certain output, for example, a document or some artifact, that is, in turn, consumed by the step that follows.

The following diagram outlines the basic steps that were introduced by the waterfall model:

• Requirement collection: During this stage, the customer's requirements are captured and analyzed and a requirements document is produced.
• Design: Based on the requirement's document contents, analysts will plan the system's architecture. This step is usually split into two sub-steps: the logical system design, which models the system as a set of high-level components, and the physical system design, where the appropriate technologies and hardware components are selected.
• Implementation: The implementation stage is where the design documents from the previous step get transformed by software engineers into actual code.
• Verification: The verification stage follows the implementation stage and ensures that the piece of software that got implemented actually satisfies the set of customer requirements that were collected during the requirements gathering step.
• Maintenance: The final stage in the waterfall model is when the developed software is deployed and operated by the customer:

One thing to keep in mind is that the waterfall model operates under the assumption that all customer requirements can be collected early on, especially before the project implementation stage begins. Having the full set of requirements available as a set of use cases makes it easier to get a more accurate estimate of the amount of time that's required for delivering the project and the development costs involved. A corollary to this is that software engineers are provided with all the expected use cases and system interactions in advance, thus making testing and verifying the system much simpler.

The waterfall model comes with a set of caveats that make it less favorable to use when building software systems. One potential caveat is that the model describes each stage in an abstract, high-level way and does not provide a detailed view into the processes that comprise each step or even tackle cross-cutting processes (for example, project management or quality control) that you would normally expect to execute in parallel through the various steps of the model.

While this model does work for small- to medium-scale projects, it tends, at least in my view, not to be as efficient for projects such as the ones commissioned by large organizations and/or government bodies. To begin with, the model assumes that analysts are always able to elicit the correct set of requirements from customers. This is not always the case as, oftentimes, customers are not able to accurately describe their requirements or tend to identify additional requirements just before the project is delivered. In addition to this, the sequential nature of this model means that a significant amount of time may elapse between gathering the initial requirements and the actual implementation. During this time – what some would refer to as an eternity in software engineering terms – the customer's requirements may shift. Changes in requirements necessitate additional development effort and this directly translates into increased costs for the deliverable.

The iterative enhancement model that's depicted in the following diagram was proposed in 1975 by Basili and Victor ^[2] in an attempt to improve on some of the caveats of the waterfall model. By recognizing that requirements may potentially change for long-running projects, the model advocates executing a set of evolution cycles or iterations, with each one being allocated a fixed amount of time out of the project's time budget:

The spiral development model was introduced by Barry Boehm in 1986 ^[5] as an approach to minimize risk when developing large-scale projects associated with significant development costs.

In the context of software engineering, risks are defined as any kind of situation or sequence of events that can cause a project to fail to meet its goals. Examples of various degrees of failure include the following:

• Missing the delivery deadline
• Exceeding the project budget
• Delivering software on time, depending on the hardware that isn't available yet

As illustrated in the following diagram, the spiral model combines the ideas and concepts from the waterfall and iterative models with a risk assessment and analysis process. As Boehm points out, a very common mistake that people who are unfamiliar with the model tend to make when seeing this diagram for the first time is to assume that the spiral model is just a sequence of incremental waterfall steps that have to be followed in a particular order for each cycle. To dispel this misconception, Boehms provided the following definition for the spiral model:

Under this definition, risk is the primary factor that helps project stakeholders answer the following questions:

• What steps should we follow next?
• How long should we keep following those steps before we need to reevaluate risk?

When we talk about agile development, we usually refer to a broader family of software development models that were initially proposed during the early 90s. Agile is a sort of umbrella term that encompasses not only a set of frameworks but also a fairly long list of best practices for software development. If we had to come up with a more specific definition for agile, we would probably define it as follows:

The popularity of agile development and agile frameworks, in particular, skyrocketed with the publication of the Manifesto for Agile Software Development in 2001 ^[3]. At the time of writing this book, agile development practices have become the de facto standard for the software industry, especially in the field of start-up companies.

In the upcoming sections, we will be digging a bit deeper into some of the most popular models and frameworks in the agile family. While doing a deep dive on each model is outside the scope of this book, a set of additional resources will be provided at the end of this chapter if you are interested in learning more about the following models.

Lean software development is one of the earliest members of the agile family of software development models. It was introduced by Mary and Tom Poppendieck in 2003 ^[10]. Its roots go back to the lean manufacturing techniques that were introduced by Toyota's production system in the 70s. When applied to software development, the model advocates seven key principles.

This is one of the key philosophies of the lean development model. Anything that does not directly add value to the final deliverable is considered as a blocker and must be removed.

Typical cases of things that are characterized as waste by this model are as follows:

• Introduction of non-essential, that is, nice-to-have features when development is underway.
• Overly complicated decision-making processes that force development teams to remain idle while waiting for a feature to be signed off – in other words: bureaucracy!
• Unnecessary communication between the various project stakeholders and the development teams. This disrupts the focus of the development team and hinders their development velocity.

The development team should never assume that the customers' requirements are static. Instead, the assumption should always be that they are dynamic and can change over time. Therefore, it is imperative for the development team to come up with appropriate strategies to ensure that their view of the world is always aligned with the customer's.

One way to achieve this is by borrowing and implementing some facets of other models, such as the iterative model we discussed in the previous section, or by tweaking their workflows accordingly so that deliverables are always built in an incremental fashion and always with an up-to-date version of the customer's requirements.

Externally acquired knowledge is, of course, only half of the equation; the development teams themselves are also another source of knowledge. As teams collaborate to deliver a piece of software, they discover that certain approaches and practices work better together than others. In particular, some approaches accelerate the team's development velocity, while others hinder it. Due to this, it is important for teams to capture this bit of tacit knowledge, internalize it, and make it available to other teams in the future. One way to achieve this is by making another team for the teams to sync up, reflect on their workflows, and discuss any potential issues.

As with all the models in the agile family, the lean model is devoid of any attempt to force project stakeholders into making all the required decisions at the beginning of the project. The reasoning behind this is quite simple: people are more likely to be convinced that change is needed when they have not already committed to a particular set of actions.

The lean model actively encourages stakeholders to defer all the important and potentially irreversible decisions until a later stage in the project's development.

One of the primary reasons for project delays is undoubtedly the accumulation of defects. Defects have a definite impact on the development team's velocity as members often need to pause their current work to chase down and fix potentially field-critical bugs that were introduced by a previous development iteration.

The lean model prompts engineering teams to aggressively focus on following agile practices such as test- or behavior-driven development (TDD/BDD) in an attempt to produce lean, well-tested code with fewer defects. The benefits of this recommendation have also been corroborated by research that's been performed by Turhan and others ^[13].

Every engineering team out there would probably agree that they would like nothing more than delivering the piece of software they are currently working on as fast as possible to the hands of the customer or the end user. The most common factors that prevent teams from delivering software fast are as follows:

• Over-analyzing the business requirements
• Over-engineering the solution to fit those requirements
• Overloading the development team

Congruent to the philosophy of lean development, teams must iterate quickly, that is, they must build a solution as simple as possible, present it to the target customer as early as possible, and collect useful feedback that's used to incrementally improve the solution in subsequent iterations.

Lean development endeavors to improve the development teams' working environment by filtering out unneeded sources of distraction that increase the cognitive load on engineers and can eventually lead to burnouts.

What's more, by discouraging micro-management and encouraging teams to self-organize, team members can feel more motivated and empowered. The Poppendiecks believe that engaged and empowered people can be more productive; ergo, they can bring more value to the team and, by extension, to the company that they are a part of.

In Lean Software Development: An Agile Toolkit, Mary and Tom Poppendieck use a stream-based analogy to describe the software development process. By this definition, each stage of the development process can be treated as a potential generator of value (a value stream) for the business. The Poppendiecks claim that in order to maximize the value that flows through the various stages of development, organizations must treat the development process as a sequence of inter-linked activities and optimize them as a whole.

This is one of the most common pitfalls that organizations fall into when attempting to apply lean thinking concepts. You have probably heard of the old adage miss the forest for the trees. Many organizations, under the influence of other lean model principles such as quick delivery, focus all their efforts on optimizing a particular aspect of their development process. To the casual external observer, this approach seems to pay off in the short term. In the long term, however, the team is vulnerable to the negative side effects of sub-optimization.

To understand how sub-optimization can affect the team's performance in the long run, let's examine a hypothetical scenario: in an attempt to iterate faster, the development team takes a few shortcuts, that is, they push out less than stellar code or code that is not thoroughly tested. While the code does work, and the customer's requirements are being met, it also increases the complexity of the code base with the unavoidable side effect that more defects start creeping into the code that is delivered to the customer. Now, the development team is under even more pressure to fix the bugs that got introduced while maintaining their previous development velocity at the same time. As you can probably deduce, by this point, the development team is stuck in a vicious circle, and certainly one that is not easy to escape from.

On the other side of the spectrum, a popular and successful example of applying the concepts of whole system optimization in the way that's intended by the lean development model is Spotify's squad-based framework. Spotify squads are lean, cross-functional, multi-disciplined, and self-organizing teams that bring together all the people who are needed to take a feature through all the stages of development, from its inception to final product delivery.

Scrum is hands-down the most widely known framework of the agile family and the go-to solution for many companies, especially the ones working on new products or the ones that actively seek to optimize their software development process. In fact, Scrum has become so popular that, nowadays, several organizations are offering Scrum certification courses. It was co-created by Ken Schwaber and Jeff Sutherland and initially presented to ACM's object-oriented programming, systems, languages, and applications conference in 1995.

As a process framework, Scrum is meant to be applied by cross-functional teams working on large projects that can be split into smaller chunks of work, where each chunk normally takes between two to four weeks – also known as a sprint in Scrum terminology – to complete.

Contrary to the other software development models we've discussed so far, Scrum does not explicitly advocate a particular design process or methodology. Instead, it promotes an empirical, feedback loop type of approach: initially, the team comes up with an idea on how to proceed based on the information available at the time. The proposed idea is then put to the test for the next sprint cycle and feedback is collected. The team then reflects on that feedback, refines the approach further, and applies it to the following sprint.

As more and more sprint cycles go by, the team learns to self-organize so that it becomes more efficient at tackling the task at hand. By improving the quality of communication between the team members while at the same time reducing distractions, teams often observe a boost in the team's output, also known as team velocity in agile terminology.

One important thing to keep in mind is that while this chapter examines Scrum from the perspective of a software engineer, the Scrum process and principles can also be applied when working on other types of projects that do not involve software development. For instance, Scrum can also be used to run marketing campaigns, hire personnel, or even tackle construction projects.

When applying the Scrum framework to a software development team, each member can be mapped to one of the following three roles:

• The Product Owner (PO)
• The Development Team Member
• The Scrum Master (SM)

The official Scrum guide ^[12], which is freely available to download online in over 30 languages, defines the PO as the key stakeholder in a project, that is, the person who maximizes the product's value resulting from the work of the development team.

The primary responsibility of the PO is to manage the project backlog. The backlog is just a formal way of referring to the list of tasks that need to be completed for a particular project and includes new features, enhancements, or bug fixes for upcoming development cycles.

The PO must always make sure that all the backlog entries are described in a clear, consistent, and unambiguous way. Furthermore, the backlog's contents is never assumed to be static but should always be assumed to be dynamic: new tasks may be introduced while existing tasks may be removed to facilitate changes to the project requirements while development is underway. This adds an extra responsibility to the role of the PO: they need to be able to respond to such changes and reprioritize the backlog accordingly.

The development team comprises a set of individuals who implement the tasks that have been selected from the backlog. According to the basic tenets of Scrum, the team should be as follows:

• It should be cross-functional, bringing people together from different disciplines and varying skill sets
• It should not pay attention to the job titles of its members and focus on the work that's performed instead
• It should be aligned toward a single goal: completing the set of tasks that the team committed to at the beginning of each sprint

The last but equally important Scrum role is that of the SM. The SM supports both the PO and the development team members by ensuring that everyone has a clear understanding of not only the team goals but also the various Scrum processes. The SM is also responsible for organizing and running the appropriate Scrum events (ceremonies) as and when required.

Scrum prescribes a sequence of events that are specially designed to aid teams in becoming more agile and boosting their performance. Let's take a brief look at the list of essential Scrum events for the purpose of software development.

The first Scrum event that we will be examining is the planning session. During planning, the team examines the items from the backlog and commits to a set of tasks that the team will be working on during the next sprint.

As you probably expect, the team needs to periodically sync up so that all the team members are on the same page with respect to the tasks that other team members are currently working on. This is facilitated by the daily stand-up, a time-boxed session that usually takes no longer than 30 minutes. Each team member speaks in turn and briefly answers the following questions:

• What was I working on yesterday?
• What will I be working on today?
• Are there any blockers for completing a particular task?

Blockers, if left unresolved, could jeopardize the team's goal for the sprint. Therefore, it is of paramount importance to detect blockers as early as possible and engage the team members to figure out ways to work around or address them.

At the end of a sprint, the team usually holds a retrospective session where team members openly discuss the things that went right, as well as the things that went wrong, during the sprint. For each problem that's encountered, the team attempts to identify its root cause and propose a series of actions to remedy it. The selected actions are applied during the next sprint cycle and their effect is re-evaluated in the next retrospective.

Kanban, whose name loosely translates from Japanese as a visual signal or a billboard, is yet another very popular type of agile framework that has been reportedly in use at Microsoft since 2004. One of the iconic features of the Kanban model is, of course, the Kanban board, a concept outlined by David Anderson's 2010 book ^[1] that introduces the idea behind this particular model.

The Kanban board allows team members to visualize the set of items that teams are working on, along with their current state. The board is comprised of a series of vertically oriented work lanes or columns. Each lane has its own label and a list of items or tasks attached to it. As items or tasks are being worked on, they transition between the various columns of the board until they eventually arrive at a column that signals their completion. Completed items are then typically removed from the board and archived for future reference.

The standard lane configuration for software development consists of at least the following set of lanes:

• Backlog: A set of tasks to be worked on by the team in the near future
• Doing: The tasks in progress
• In review: Work that has been put up for review by other team members
• Done: Items that have been completed

It is only logical that each team will customize the lane configuration to fit their particular development workflow. For example, some teams may include an in test column to keep track of items undergoing QA checks by another team, a deployed column to track items that have been deployed to production, and even a blocked column to specify tasks that cannot proceed without the team taking some type of action.

I am sure that most of you will probably already be familiar with the physical implementation of a Kanban board: a dedicated spot on the office wall filled with colorful post-it notes. While local teams tend to enjoy having the board on a wall as it makes it quite easy to see what everyone is working on or to identify blockers just by walking by the board, this approach obviously cannot support partially or fully remote teams. For those use cases, several companies are offering the online, digital equivalent of a Kanban board that can be used instead.

DevOps is the last software development model that we will be examining in this chapter. Nowadays, more and more organizations endeavor to scale out their systems by transitioning from monolithic to service-oriented architectures (SoA). The basic premise behind the DevOps model is that each engineering team owns the services they build. This is achieved by fusing development with operations, that is, the aspects involved in deploying, scaling, and monitoring services once they get deployed to production.

The DevOps model evolved in parallel with the other agile models and was heavily influenced by the principles put forward by the lean development model. While there is no recommended approach to implementing DevOps (one of the reasons why Google came up with SRE in the first place), DevOps advocates tend to gravitate toward two different models:

• Culture, Automation, Measurement, and Sharing (CAMS)
• The three ways model

CAMS was originally invented by Damon Edwards and John Willis. Let's explore each one of these terms in a bit more detail.

As with other agile models, corporate culture is an integral part of DevOps methodology. To this end, Edwards and Willis recommend that engineering teams extend the use of practices such as Scrum and Kanban to manage both development and operations. Culture-wise, an extremely important piece of advice that Edwards and Willis offered is that each company must internally evolve its own culture and set of values that suit its unique set of needs instead of simply copying them over from other organizations because they just seem to be working in a particular context. The latter could lead to what is known as the Cargo Cult effect, which eventually creates a toxic work environment that can cause issues with employee retainment.

The second tenet of the CAMS model is automation. As we discussed in a previous section, automation is all about eliminating potential human sources of errors when executing tedious, repetitive tasks. In the context of DevOps, this is usually accomplished by doing the following:

• Deploying a CI/CD system to ensure that all the changes are thoroughly tested before they get pushed to production
• Treating infrastructure as code and managing it as such, that is, storing it in a version control system (VCS), having engineers review and audit infrastructure changes, and finally deploying them via tools such as Chef (https://www.chef.io/), puppet (https://puppet.com/), Ansible (https://www.ansible.com/), and Terraform (https://www.terraform.io/)

The letter M in CAMS stands for measurement. Being able to not only capture service operation metrics but also act on them offers two significant advantages to engineering teams. To begin with, the team can always be apprised of the health of the services they manage. When a service misbehaves, the metrics monitoring system will fire an alert and some of the team members will typically get paged. When this happens, having access to a rich set of metrics allows teams to quickly assess the situation and attempt to remedy any issue.

Of course, monitoring is not the only use case for measuring: services that are managed by DevOps teams are, in most cases, long-lived and therefore bound to evolve or expand over time; it stands to reason that teams will be expected to improve on and optimize the services they manage. High-level performance metrics help identify services with a high load that need to be scaled, while low-level performance metrics will indicate slow code paths that need to be optimized. In both cases, measuring can be used as a feedback loop to the development process to aid teams in deciding what to work on next.

The last letter in the CAMS model stands for sharing. The key ideas here are as follows:

• To promote visibility throughout the organization
• To encourage and facilitate knowledge sharing across teams

Visibility is quite important for all stakeholders. First of all, it allows all the members of the organization to be constantly aware of what other teams are currently working on. Secondly, it offers engineers a clear perspective of how each team's progress is contributing to the long-term strategic goals of the organization. One way to achieve this is by making the team's Kanban board accessible to other teams in the organization.

The model inventors encourage teams to be transparent about their internal processes. By allowing information to flow freely across teams, information silos can be prevented. For instance, senior teams will eventually evolve their own streamlined deployment process. By making this knowledge available to other, less senior, teams, they can directly exploit the learnings of more seasoned teams without having to reinvent the wheel. Apart from this, teams will typically use a set of internal dashboards to monitor the services they manage. There is a definite benefit in making these public to other teams, especially ones that serve as upstream consumers for those services.

At this point, it is important to note that, in many cases, transparency extends beyond the bounds of the company. Lots of companies are making a subset of their ops metrics available to their customers by setting up status pages, while others go even further and publish detailed postmortems on outages.

The three ways model is based on the ideas of Gene Kim, Kevin Behr, George Spafford ^[8], and other lean thinkers such as Michael Orzen ^[9]. The model distills the concept of DevOps into three primary principles, or ways:

• Systems thinking and workflow optimization
• Amplifying feedback loops
• Culture of continuous experimentation and learning

Systems thinking implies that the development team takes a holistic approach to software: in addition to tackling software development, teams are also responsible for operating/managing the systems that the software gets deployed to and establishing baselines for not only the target system's behavior but also for the expected behavior of other systems that depend on it:

The preceding diagram represents this approach as a unidirectional sequence of steps that the engineering team executes to deliver a working feature to the end user or customer in a way that does not cause any disruption to existing services. At this stage, the team's main focus is to optimize the end-to-end delivery process by identifying and removing any bottlenecks that hinder the flow of work between the various steps.

Under the first principle, teams attempt to reduce the number of defects that flow downstream. Nevertheless, defects do occasionally slip through. This is where the second principle comes into play. It introduces feedback loops that enable information to flow backward, as shown in the following diagram, that is, from right to left. By themselves, however, feedback loops are not enough; they must also serve as amplification points to ensure that the team members are forced to act on incoming information in a timely fashion. For example, an incoming alert (feedback loop) will trigger a person from the team who is on call to get paged (amplification) so as to resolve an issue that affects production:

The final principle, and the one that most agile models are imbued with, has to do with fostering a company culture that allows people to pursue experiments and improvement ideas that may or may not pan out in the end as long as they share what they've learned with their colleagues. The same mindset also applies when dealing with incidents that have adverse effects on production systems. For instance, by holding blameless postmortems, the team members can outline the root causes of an outage in a way that doesn't put pressure on the peers whose actions caused the outage and, at the same time, disseminate the set of steps and knowledge that were acquired by resolving the issue.

Through the course of this chapter, we briefly discussed the different types of roles that you may encounter when working with companies of various sizes, as well as the special set of skills that each role depends on.

We started by examining a broad list of popular models, methodologies, and frameworks for delivering software, ranging from the traditional models that advocate a top-bottom approach (waterfall, iterative enhancement) to agile models that are better suited for the fast-paced and constantly changing environment that contemporary organizations operate in.

By reaching the end of this chapter, you should have acquainted yourself with the pros and cons of each model and the situations where each model should be applied. I sincerely hope that this knowledge will prove useful the next time you need to decide on which software development model to use for your next project.

Last but not least, we should always keep in mind that the cornerstone of the software engineering process is the actual production of high-quality software! In the next chapter, we will put on our engineering hat and discuss approaches for writing clean, organized, and easy-to-maintain code using Go.

1. What is the definition of software engineering?
2. What are some of the questions that every SWE should be able to answer?
3. Compare the role of an SWE and an SRE. What are the key differences between the two roles?
4. Name some of the deficiencies of the waterfall model. Explain how the iterative enhancement model attempts to address those deficiencies.
5. What are the most common sources of waste according to the lean development model?
6. Provide an example where focusing all the optimization efforts on a single step of the development process can have a negative effect on the efficiency of the end-to-end process.
7. What are the key responsibilities of the PO and the SM in the Scrum framework?
8. What is the role of retrospectives in Scrum? What topics should the team be discussing and what should be the expected outcome of each retrospective session?
9. Why are automation and measuring important when following the DevOps model?
10. You are working for ACME Gaming System, a company with the vision to disrupt the already mature and highly competitive gaming console market. To this end, the company has partnered with Advanced GPU Devices, a well-known graphics chip manufacturer, to prototype a new GPU design that should allow the upcoming gaming console to blow competitor consoles out of the water. Your task, as the project's lead engineer, is to design and build the software that will power the new console. Which software development model would you choose? Explain the reasoning behind your decision.

1. Anderson, David: Kanban: Successful Evolutionary Change in Your Technology Business: Blue Hole Press, 2010 — ISBN 0984521402 (https://www.worldcat.org/title/kanban-successful-evolutionary-change-in-your-technology-business/oclc/693773272).
2. Basili, R.; Turner, J.: Iterative Enhancement: A Practical Technique for Software Development. In: IEEE Transactions on Software Engineering Vol. 1 (1975), pp. 390–396.

3. Beck, Kent; Beedle, Mike; Bennekum, Arie van; Cockburn, Alistair; Cunningham, Ward; Fowler, Martin; Grenning, James; Highsmith, Jim; et al.: Manifesto for Agile Software Development.
4. Beyer, Betsy; Jones, Chris; Petoff, Jennifer; Murphy, Niall Richard: Site Reliability Engineering: How Google Runs Production Systems. (https://landing.google.com/sre/sre-book/toc/index.html) (https://www.worldcat.org/title/site-reliability-engineering/oclc/1112558638).
5. Boehm, B: A Spiral Model of Software Development and Enhancement. In: SIGSOFT Softw. Eng. Notes Vol. 11. New York, NY, USA, ACM (1986), Nr. 4, pp. 14–24.
6. Constantine, L.: Beyond Chaos: The Expert Edge in Managing Software Development: Addison-Wesley Professional, 2001 — ISBN 9780201719604 (https://www.worldcat.org/title/beyond-chaos-the-expert-edge-in-managing-software-development/oclc/46713128).
7. IEEE: IEEE Standard Glossary of Software Engineering Terminology: IEEE; IEEE, 1990.
8. Kim, G.; Behr, K.; Spafford, G.: The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win: IT Revolution Press, 2018 — ISBN 9781942788294 (https://www.worldcat.org/title/phoenix-project-a-novel-about-it-devops-and-helping-your-business-win/oclc/1035062278).
9. Orzen, M. A.; Paider, T. A.: The Lean IT Field Guide: A Roadmap for Your Transformation: Taylor and Francis, 2015 — ISBN 9781498730389 (https://www.worldcat.org/title/lean-it-field-guide/oclc/1019734287).
10. Poppendieck, Mary; Poppendieck, Tom: Lean Software Development: An Agile Toolkit. Boston, MA, USA: Addison-Wesley Longman Publishing Co., Inc., 2003 — ISBN 0321150783 (https://www.worldcat.org/title/lean-software-development-an-agile-toolkit/oclc/868260760).
11. Royce, W.: Managing the development of large software systems: concepts and techniques. In: Proc. IEEE WESTCON, Los Angeles (1970), pp. 1–9. — Reprinted in Proceedings of the Ninth International Conference on Software Engineering, March 1987, pp. 328–338.
12. Schwaber, Ken; Sutherland, Jeff: The Scrum Guide (2014).
13. Turhan, Burak; Layman, Lucak; Diep, Madeline; Shull, Forrst; Erdogmus, Hakan: How Effective is Test-Driven Development? In: Wilson, G.; Orham, A. (publisher): Making software: what really works, and why we believe it: ISBN 978-0596808327 (https://www.worldcat.org/title/making-software-what-really-works-and-why-we-believe-it/oclc/836148043), S. 207–219.