The underlying concepts of Scrum

There are three categories of concepts that provide the foundation for Scrum. The first set of ideas resides in management approach for complex and chaotic situations, the second is a set of core values, and the third is a deep-seated focus on delivering product in a lean fashion. You should gain a thorough understanding of these concepts, not for passing a certification exam, but for explaining to others in the organization why and how Scrum is different than waterfall.

Complex adaptive systems

Complex adaptive systems are made up of varied interdependent elements whose relationships change as a result of experience. Complex systems adapt; as one variable emerges, others are affected. A simple example from biology is the Staph bacteria; in the 1940s, more than 99 percent of Staph bacteria were sensitive to penicillin. Today, Staph is resistant to penicillin; due to the introduction and exposure to the antibiotic, the bacteria evolved to survive.

In our world, requirements, technology, and people are intertwined; a change in one means a change in the other. When I was a kid, I loved to go to the movie theater to, and sometimes I would go just to play video games at the adjoining arcade (Ms. Pac-Man, Frogger, Pole Position, and Asteroids). My family couldn't afford a home system such as an Atari (and, having played at the arcade, I much preferred the arcade to an Atari home system anyway). Well, all that changed when Nintendo launched its Nintendo Entertainment System (NES). The kids next door got the system as a Christmas gift that year and next thing you know, we were over at their house many times per week after school playing Super Mario Brothers. Then along came the handhelds, Super NES, PlayStation, Sega to name a few, and now mobile phones and interactive play are all the rage. The advances in 16-bit and 32-bit consoles initially improved the user's experience at home with better graphics, competitive pressure resulted in more innovative home products from which a user could choose, and new technologies have allowed gaming companies to extend offerings into new arenas and find new markets. The technology, user needs and requirements, and developers in the gaming industry are all intertwined. A change in one necessitates a change in the others. Little did I realize, at that time, Mario and Luigi's importance to technology—much greater than just saving the princess!

Here's the deal: today's users change their minds about needs and desires minute to minute, developers continuously release new technologies and better versions of the old, and stakeholders and customers are never shy about saying how they feel about existing features. This emergence, that is, the evolution of new knowledge, needs, and desires about requirements and technologies during the course of a project, contributes to one heck of a complex situation. Complex projects cannot be adequately managed in a traditional approach, where all tasks are defined up front and then delegated to workers. In situations where the exact answer is not known, and must be discovered, an empirical process or evidence-based process is the best approach to use. (Take a look at the article A Leader's Framework for Decision Making, by David J. Snowden and Mary E. Boone, at Harvard Business Review). But it's more than just applying the right tool or technique to the matter; complex systems involve complex human interactions. While a complex project requires a process that allows for emergence, it also requires a leadership style that facilitates humans getting along together.

The following figure is the famous Stacey Matrix, adopted from Dr. Ralph Stacey, Professor of Management at the Business School of the University of Hertfordshire, who has spent many years exploring how the complexity sciences can be leveraged to understand organizations. You may remember hearing about his work during your ScrumMaster training. The Stacey Matrix looks at certainty and agreement, which correlate to technology and requirements, respectively, in our world. The farther from certainty and agreement a situation is, the more complex or even chaotic the situation; on the other hand, a simple situation is one with agreement and knowledge. Think about an easy project from your career. The requirements for the new feature were very clear and straightforward, and you wrote all the existing code a few weeks ago. It was easy for you to provide a time estimate since you figured that surprises weren't likely. This is a simple situation, rare in the software world. A complex scenario, on the other hand, is one in which you may not have been very familiar with the code, the customer often changed her mind, and the vendor team never met its deadlines. In this case, it was impossible for you to estimate beyond the tip of your nose as surprises (mostly bad) were guaranteed.

You'll notice that I overlaid the Scrum roles and control points onto the Stacey Matrix. ScrumMasters know that they do not control complex projects by acting as task masters and telling everyone what to do every day; rather, control is acquired by allowing a team to self manage through a series of sprints (or time-boxes) and by requiring the product owner to force rank the product backlog. Each role in Scrum has a specific and important set of responsibilities, each steeped in the idea of controlling chaos (in fact, Ken Schwaber's original Scrum website is called www.controlchaos.com). Many people ask me if there is a role called project manager in Scrum and the answer is a resounding no! The reason is that by giving control to only one person over what gets done, how it gets done, and why it gets done results in chaos and other bad effects. Three main responsibilities divided among three main roles creates checks and balances in our complex project systems; these three main Scrum roles, coupled with a simple process framework, control chaos. Each sprint, by focusing on a narrow selection of product backlog, an empowered team devises the appropriate technical solution bit by bit, thus bringing the project closer to the simple range sprint after sprint. If these controls are not protected, it is all too easy for the scale to unfavorably tip. Scrum is a very simple concept and extremely powerful once implemented (and yet rather difficult to do). You, my dear ScrumMaster, must keep that sprint to the simpler side of things by thwarting interruptions and helping the product owner keep the product backlog in order.

Even though Dr. Stacey's matrix and Snowden and Boone's article provide us with nice frameworks of thought to choose the right process for the situation, please do not oversimplify this information! Dr. Stacey didn't publish the matrix beyond the second edition of his book, and he's now up to six editions! What happened? Dr. Stacey stopped publishing the matrix because people oversimplified it, thinking that one can just choose a tool or technique depending on the classification of their project. Dr. Stacey warns us that it's not that simple, that, "what happens is the result of the interplay between the intentions and strategies of all involved and no one can control this interplay, hence the fundamental uncertainty and unpredictability of human life." This is another way of saying that Scrum doesn't fix the problems, people do; Scrum only exposes where the challenges lie. People must solve the problems exposed by Scrum and should craft different solutions for different scenarios. This book attempts to explore how the ScrumMaster may facilitate some of those challenges both inside and outside of the team.

Knowledge is the outcome of experience. In a traditional approach to managing projects, all the decisions are made at the beginning of the project. This has always been interesting to me because a person has the most knowledge about a project at the end of the project! Scrum puts urgency on teams so that they deliver value quickly; in doing so, knowledge about requirements and technology increases rapidly and team dynamics begin to emerge and stabilize. The sooner a team gets started, the better. When a team has knowledge its members can make better, well-informed decisions. Scrum teams delay decisions around lower-value requirements for a later point in time, as requirements in their fast-paced technology worlds are bound to change anyway.

The empirical process control barstool

There are four legs supporting the Scrum barstool that make it unique and effective in complex situations: prioritized product backlog, dedicated cross-functional team, time-boxes, and inspection and adaptation. These four entities form the necessary boundaries that allow us to call Scrum an empirical process. When a barstool loses a leg, it is imbalanced (ever try sitting on one?); even if one leg is slightly shorter than the others, the barstool is wobbly and annoying. The same idea applies with the Scrum concepts: when the team become lax on one or more boundaries, the project gets loosey-goosey; the experiment gets shaky as it loses its control points. For example, if for some reason the ScrumMaster allows interruptions during a sprint, the team runs the risk of not finishing anything by the sprint's end. Everyone must stop and start again—we call this chaos. Likewise, if the product owner does not rank features in the product backlog, and no thought is given to priority, the team may or may not deliver the right features to customers. A simple violation of these guiding principles can cause major downstream effects. If a team adheres to these four principles of Scrum, it will likely attain a higher degree of stability and knowledge.

Ken Schwaber and Jeff Sutherland designed these four "legs", or principles, of Scrum to enable empirical process control for software development projects (although the techniques are useful in just about any type of project). Scrum is an evidence-based way of progressing through a project or initiative. Without designated points in time around which people can provide feedback on features, like in waterfall projects, the team, management, and customers really have no idea what is being developed. Try to recount a status meeting for a traditionally managed project. What was that meeting like for you? I can say that, for me, it was often daunting. I would present a status report with a bunch of red, green, and yellow dots on it, some numbers and percentages of 'phase complete', and sometimes a list of risks. I can honestly say that as a traditional project manager, if I didn't truly know the status, I'd just put a yellow dot on the dashboard, reckoning I'd figure it out later. Or, even worse, I'd stay at work very late for a few nights ahead of the status meeting attempting to make the Gantt chart look something like reality. Scrum, with its emphasis on potentially shippable product increments each and every sprint leaves nowhere to hide. Sprint reviews replace status meetings; and the status is real—that is, the team shows real features that work! That is exciting and tangible! It's evidence, and we can work with evidence! Instead of a bunch of guesswork about what we think is the status of the project, in Scrum we can actually see, touch, and feel the status of the project by attending a sprint Review and interacting with the software or system as it exists at that point in time.

I recently watched a television program that featured George Cleverley and Company, a famous London shoemaker. Founded in 1958, Cleverley makes some of the most expensive handmade shoes in the world, shodding some of the world's most discriminating tootsies. In order to make such a beautiful product of unprecedented quality, the shoemakers must work very closely with the customer to understand their desires and requirements. After crafting precise shoe-lasts (forms around which the shoe is built) made from several measurements of a customer's foot (bunions and corns included), the shoemaker must then work with the customer's choices regarding various leathers, soles, dyes, laces, thistles, welts, and stitching in order to make the shoe that the wearer has envisioned and will love. It takes months for a customer's shoes to be finished, and at least three thousand U.S. dollars, but for 58 years customers have gladly paid and anxiously waited for the call that their shoes are ready. George Cleverley and Company involves its customers in the design and creation; there's simply no other way to build the perfect shoe. In reading about the craftsmanship, attention to detail, and customer satisfaction—the hallmarks of the Cleverley brand—I realized that the process in which Cleverley shoes are built is very Scrum-like: an iterative creation process where the customer is an essential part of the desired outcome. Even though the product backlog in this process is relatively fixed—that is, the order of the steps in shoemaking is fixed, the customer must be involved every step of the way.

Scrum core values

In addition to the basics in complex adaptive systems theory, Scrum also has at its center five core values: commitment, focus, openness, respect, and courage. (Take a look at Agile Software Development with Scrum, by Ken Schwaber, Prentice Hall.) We will explore these in more detail in Chapter 8, Everyday Leadership for the ScrumMaster.

Teams commit to their goals for the sprint, product owners commit to ordering the product backlog, and ScrumMasters commit to removing obstacles along the way in order to steady the flow of product development. The Scrum team should do whatever is necessary in order to meet their goals, and it's important that they are empowered to do so.

Additionally, for a team to be able to complete its work, its members must be allowed to focus. The ScrumMaster does not allow changes in the sprint's commitment so that the team may keep its focus. When a team gives its full attention to the problem, its work is much more productive, predictable, and fulfilling.

As Scrum uses empirical process control to make progress through a project, it is essential that the results and experience of an experiment (that is, a sprint) are visible. For example, the sprint review provides visibility into how the product's features and capabilities have emerged, and the sprint retrospective engages the team members to discuss their personal and team successes and challenges experienced during the sprint. Once visibility exists, inspection and adaptation can occur. Again, evidence (information) is the heart of an empirical process; thus, openness is one of the five core values.

Respect is something that all humans should have for each other, but unfortunately it does not always exist. In order to be its best, a team's members need to respect for each other and, the knowledge that each brings to the table, experiences, working styles, and personalities, just to name a few. Respect doesn't come for free; it is earned. Scrum team members should be dedicated, cross-functional, empowered, and self-organizing—any team that is not provided this environment or structure will have a tough time achieving mutual trust and respect. We will discuss in subsequent chapters how a ScrumMaster can model (and thus create) trust and respect within the team.

It takes buckets of courage for a ScrumMaster to apply Scrum the way it was intended. Even though Scrum is very simple in design, people end up over-engineering its intended plainness. I often see management at various companies bring in Scrum as a way to do more with less (not always the case!), and they don't at the very least realize or acknowledge that the use of Scrum puts pressure on organizations to change. One of the primary responsibilities of the ScrumMaster is to help the organization identify its weaknesses so that it may improve. This takes courage. Sometimes, a team has to push back on the product owner when asked to take on too much during a sprint. It takes courage to say no to that sort of pressure. A product owner must have courage when communicating with other stakeholders about the reality of a project. These are just a few examples that require courage in order to not break the boundaries of empirical process or introduce more waste into the product or process.

An interesting tidbit: the Scrum core values sound a lot like the U.S. Army's seven values of loyalty, duty, respect, selfless service, honor, integrity, and personal courage. We might not find that too surprising as the grandfathers of Scrum have military backgrounds: Jeff Sutherland was a Top Gun fighter pilot in the U.S. Air Force, and Ken Schwaber attended the U.S. Merchant Marine Academy.

Scrum is inherently lean

Lean software development is the result of the translation of lean manufacturing into the software and systems development space. Whether it falls underneath the Agile umbrella is hotly debated in the community, yet it really doesn't matter when it's all said and done. This is because the seven principles of lean are inherently reflected in Agile methods, but perhaps in different ways and with different lexicons or naming conventions.

The first principle of Lean is to eliminate waste. From unclear requirements, unused documentation, hand-offs, wait time, and so forth, lean practitioners seek to reduce or eliminate altogether these wastes from the process. The Scrum framework echoes this lean sentiment by providing the retrospective so that a team may discover and fix anything that's not working well. You'll often hear teams discussing subjects such as wasteful documentation, overbearing and/or manual processes, too many defects, and other such issues in the retrospective.

Lean thinking also stresses that everyone on the team gets the chance to learn. Scrum mirrors this by requiring that every sprint ends with a potentially shippable product increment. This quality threshold requires that team members code, integrate, and perform tests, of various sorts in order to learn what is defective, while the product owner learns about the emerging product. Additionally, members of a team work very closely together, which means that they often learn a little about the others' domain.

In any project, people know the most about the project at the end of the project. Scrum teams would rather make well-informed decisions; therefore, they do not make decisions about every requirement up front. This is a manifestation of the fourth value of Lean: decide as late as possible.

Lean says deliver fast; in Scrum we deliver at most every 30 days, while many teams deliver even faster than that. Lean says empower the team, and so does Scrum. Lean says that integrity should be built into the system; Scrum answers this by requiring a team to define done with a customer. Finally, Lean guides us to see the whole—how the entire value stream, or chain of events leading to customer value, operates. Any bottlenecks should be removed immediately, and teams should be staffed so that they can complete finished product increments. Scrum reflects this by guiding us to create dedicated, cross-functional teams that conduct retrospectives. Such retrospectives help us unearth bottlenecks (or obstacles as called in Scrum) so that they can be eliminated.