Understanding when testing should begin

There is no clear line between development and testing, even when testing is a dedicated role within a company. It might seem obvious that first, you design your intended product, then you build it, and then you test it, as shown in the following diagram:

Figure 1.1 – An idealized waterfall model

This is known as the waterfall model of development. It has the advantage of having a clear point where testing should begin – that is, when the development team hands over the code. However, while such strict and simple handovers might work for bridges and houses, software development has more complexity – it has to handle many different inputs, and has the opportunity to be far more flexible.

In Agile approaches to software development, development cycles have multiple interacting feedback loops that contribute to the final product. As a tester, you begin your work at some point within those cycles. While testing could wait until the developers are perfectly happy with their product, in practice, this adds too much delay. Getting testers involved early in a project has many advantages:

• Helps the testers understand the feature
• Gives testers time to prepare tools or datasets that they need
• Lets testers provide feedback on the design process
• Lets testers begin testing earlier (than usual), even if only in a limited way

This is a part of shift-left testing to involve testers early in projects, which is a worthy goal but is challenging as there is no clear place to start. There is no day when a task is complete and testing should begin. Instead, testers need to gradually start testing based on the available specifications and code.

Another challenge for early testing is to integrate with other methods of verification. The developers will (hopefully) perform manual checks on their code and write unit tests for individual functions. This will already produce bugs and suggestions that feed back into the code implementation and the feature specification, as shown in the following diagram:

Figure 1.2 – Interactions and feedback in an Agile development model

In an Agile model, the specification still guides the implementation and suggests tests for developers and testers to run. But now, there can be far more feedback – the specification can change based on technical limitations and possibilities, then the developers will fix their bugs, and some developer tests may uncover changes needed in the specification.

While the waterfall model had a specific flow in time – in Figure 1.1, from left to right – in an Agile model, there is a constant back and forth between the different tasks. They are performed almost simultaneously. The specification is still a necessary first step to state what the developer should implement, but after that, the phases largely overlap. In test-driven development (TDD), for example, the testing will come before the implementation.

Notice that system testing performed by a separate test function isn’t included in this diagram yet. The only testing described there is performed by the developers themselves, which is an important ingredient in the testing mix. The system testing described in this book extends and completes the testing started by the developers.

Crucially, in an Agile project, this flow won’t happen to an entire feature in one go. Instead, the feature is broken down into multiple parts, each of which might be developed in parallel by different team members:

Figure 1.3 – Different parts of a feature developed in parallel within an Agile model

Instead of a feature being fully specified and fully implemented, as shown in the waterfall model in Figure 1.1, Agile recommends splitting tasks into the smallest possible functional units so that work can proceed on each unit in parallel. In this model, all those interactions happen simultaneously for different parts of the project as they are implemented and developers start their testing. This parallel working, with the opportunity for feedback at multiple levels, give Agile projects great flexibility.

The situation becomes even more complicated than this, of course, since the lessons you learn while implementing part 2 of a feature can feed back into part 1, whether from its specification, implementation, or initial testing, as shown in the following diagram:

Figure 1.4 – Interaction between different parts of an Agile feature development

While writing the specification for part 2, you may identify changes you need in the specification for part 1. Likewise, the implementation or developer testing of part 2 might need a specification change. The implementation and developer testing of part 1 might also benefit from insights learned while implementing part 2 so that there is constant feedback between those development tasks.

There are several lines of interaction not drawn in Figure 1.4. The developer testing also feeds back into the specification, and the work in part 3 of the feature feeds back into the tasks in part 1, and so on. I left those lines off the preceding diagram for sanity’s sake.

Where, in that mess of interactions, should system testing start? Where should it fit into this development flow overall? At any time, any part of the feature will be partially specified, partially implemented, and partially covered by development tests.

Unlike the waterfall model, there is no clear starting point where system tests should begin. However, there is a place where system tests can fit into this development cycle, which we can see if we simplify the diagram by considering a single feature. System testing should build from the developer testing while being guided and providing feedback to all previous stages, as shown in the following diagram:

Figure 1.5 – System testing as part of an Agile development model

Considering a single feature, the system test design can begin as soon as the specification has been started. Once there are requirements, you can develop the tests to check them. Once the implementation has begun, you can design tests using white-box techniques (see Chapter 6, White-Box Functional Testing) to ensure you have checked every code path. And the system tests should extend and complement the testing the developers perform.

The system testing should also provide feedback on the other tasks in the development cycle. Most obviously, bugs found in system testing need to be fixed in the implementation. Some bugs will lead to specification changes, especially clarifications or descriptions of complex situations that weren’t initially described. System tests also interact with the developer testing. They should avoid overlap but cover the gaps in unit tests, and some tests identified at the system test level might be best implemented as unit tests and should be moved there.

Remember, the preceding diagram doesn’t show the tasks in a timely order. Just because system testing appears on the right doesn’t mean it is performed last. Like the implementation and developer testing, it can start as soon as the first parts of the feature specification are ready. You can design tests against those specifications before any work by the developers starts and you can provide feedback on that specification.

Finally, we can put all this together, showing the interactions as multiple parts of a feature are developed and tested simultaneously:

Figure 1.6 – Interaction between different parts of a feature under development, including system testing

As the preceding diagram shows, there is a lot to manage, even at this high level of detail. This book will show how to write great feature specifications (see Chapter 2, Writing Great Feature Specifications, and Chapter 3, How to Run Successful Specification Reviews) and how to build on the developer’s implementation and testing (Chapter 6, White-Box Functional Testing). In addition, you need to consider the details of the system tests themselves, which will be covered in the remainder of this book.

Next, we’ll consider the main tasks involved in system testing and where to begin.