Problems with manual software development life cycle practices

Now that you have the general picture of what happens to software between the time that developers have finished writing it and the time that users can get their hands on it, you can start to understand how tough this process can be. Many tasks need to happen along this path of delivering secure, working code to users:

Figure 1.4 – Major tasks in the SDLC

Some of these tasks are normally done manually, while others can be automated either partially or fully. But both approaches have problems associated with them, which turn each task into a potential pain point.

What are the difficulties of manually performing these tasks? Let’s take a look:

• They take time. They often take significantly more time than you budget for them, even after you’ve had experience manually performing them in the past. There are countless ways things can go wrong when performing any of these tasks, all of which require time-consuming troubleshooting and remediation. Even when everything goes right, there is simply a lot of work involved with each of these tasks. And remember the 1979 law proposed by physicist Douglas Hofstadter: It always takes longer than you expect, even when you take into account Hofstadter’s Law.
• They are error-prone. Because you’re relying on humans to perform them – humans who might be tired, bored, or distracted – they’re all susceptible to misconfigurations, data entry mistakes, or steps that have been forgotten or applied in the wrong order, to name just a few ways human error can lead to things going wrong.
• They are tough on employee morale. Nobody likes doing routine, repetitive work, especially when the stakes are high and you must get it right. The prospect of running through a standard 2-hour set of manual tests for the 20th time in 2 weeks has caused many a QA engineer to wonder if maybe software wasn’t the smartest career choice for them after all.
• They have a high potential for miscommunication or misreporting. When a manual tester has finished running their stultifying 2-hour test suite, do they have enough brain cells left to accurately record what worked and what didn’t? All that testing is pointless if we can’t rely on the results being recorded accurately, but anyone who has executed a difficult manual test plan knows how many ambiguities there can be in the results, how many unexpected conditions there can be to potentially skew the tests, and how hard it can be to know how to explain these factors to the people who rely on that reporting. And that’s not even factoring in the very large possibility of simply recording results incorrectly, even when they’re unambiguous.

For all these reasons, you can see how expensive – in terms of time, money, and employee goodwill – manual tasks are likely to be.

But if some of the tasks we’ve described can be automated, would that eliminate the problems that we face with manual tasks? Well, it would solve some problems. But adding a series of automation tools to the SDLC would introduce a whole new set of problems. Consider all the extra effort and expense involved with doing so and all the tasks that building custom toolchains entail:

• Researching and selecting tools for each automatable task.
• Buying and renewing licenses for each tool
• Choosing a hosting solution for each tool
• Provisioning users for each tool
• Learning different GUIs for each tool
• Managing databases and other infrastructure for each tool
• Integrating each tool with other tools in the SDLC
• Figuring out how to display the status and results of each tool in a central location if that’s even possible
• Deal with tools that are buggy, that become deprecated, or that become less compelling as better alternatives appear on the market

Even after all the problems of manual or automated tasks are handled, there’s one big problem that’s unavoidable for teams that use this model: it’s a sequential workflow. Steps happen one after another. One team writes the software, then throws the code over the wall to another team that builds the software. That team, in turn, tosses the code to a third team that’s responsible for validating the software. When they’re done, they generally send it to yet another group of engineers who do security testing. Finally, a release team gets ahold of it so that they can deploy the code to the right place. There are plenty of ways this process can deviate from this basic description, but the fundamental concept of doing one step at a time, passing the code to the next step only after the earlier steps are complete, is a trait that many, many software development teams’ workflows shared.

So far, it might not be obvious why sequential workflows pose a problem, so let’s spell it out. Because of the difficulty of executing these steps manually, or the hassles of keeping automated steps running smoothly and reliably across multiple tools, this workflow typically happens only sporadically. How often the code gets run through these steps varies from team to team, but the time and expense involved means that code changes typically stack up over days, weeks, or sometimes even months before they get built, validated, secured, and deployed properly. And that, in turn, means that problems detected during this process are expensive to fix. If a functional test fails, a security test detects a vulnerability, or an integration test reveals that the code doesn’t play well together once it’s all deployed to the same environment, identifying what code is causing the problem is like finding a needle in a very large haystack. If 5,000 lines of code across 25 classes have changed, 16 dependencies have been upgraded to more recent versions, the Java version has changed from version 16 to version 17, and the test environment is running on a different version of Ubuntu, those are a whole lot of variables to investigate when you’re tracking down the source of the problem and figuring out how to fix it.

At this point, you know enough about traditional, pre-DevOps software development that we can summarize the biggest problem it faces in one sentence: sequential workflows that involve manual tasks or automated tasks performed by different tools cause development to be slow, releases to be infrequent, and the resulting software to be of less high quality than it could be.

But there’s good news: DevOps was invented to solve these problems. And GitLab CI/CD pipelines were invented to make DevOps easier to use. We’ll look at both of those things next.