Running complete exploratory testing

Exploratory testing is a smaller version of all the following testing, pulling out a few of the most important tests from each section. It is introductory testing, which means it briefly covers many areas in the same way that this chapter has introduced this book. Exploratory testing should cover the following aspects:

• Black-box functional testing
• White-box functional testing
• Error cases
• User experience testing
• Security testing
• Tests for maintainability
• Non-functional testing

With less time available for exploratory testing than in the complete test plan, you should prioritize these aspects carefully.

As we’ve seen already, exploratory testing starts with black-box functional testing, using no knowledge of the underlying implementation, and concentrating only on the working cases. This should form the majority of the work you do here.

While there are advantages to naivety about how a feature is implemented, even during exploratory testing, it is helpful to know some details of its architecture and design. If one minor aspect of the feature requires a large section of code, then you should test that aspect far more than its use might suggest. If there’s more code required for its implementation, then there is more to go wrong and a higher chance of defects. So, even while exploring, it’s essential to do some white-box testing, informed by knowledge of the feature’s design. This should come after black-box testing so that you can make the most of your lack of assumptions about its behavior first.

You can also start trying error cases during exploratory testing, for instance, by deliberately leaving fields blank, entering invalid information, or attempting to trigger errors. This shouldn’t be the focus – making sure the feature works comes first. But even this early, you can probe the behavior with incorrect inputs.

As described previously, exploratory testing is a great time for finding usability issues when a feature is new to you before you learn how it works, inside and out. Feedback on usability should be a key deliverable from all exploratory testing.

You can also test security during the exploratory phase. Again, you need to prioritize these tests – the most obvious attacks can be quick to run SQL injection and scripting attacks or attempting to access information without the necessary permissions. Are the required certificates in place, and is the traffic encrypted? See Chapter 9, Security Testing, for more details on how to run those kinds of tests. Major deficiencies such as those can be easily spotted and raised early. Security shouldn’t be a focus for exploratory testing compared to functional and usability testing, but this is where it can start.

Exploratory testing can also start to examine the maintainability of the code. How useful are the logs? Do they record the application’s actions? Are the relevant events generated? What monitoring is in place for this service? Early in the project, in the first version of the code, the answer might be as simple as noting that events are not ready yet, or the gaps within them. This is the time to start writing the list of those requirements. Maintainability can be low on the priority list for a project, so it’s important to note the requirements early.

Exploratory testing does not typically cover non-functional testing since that often requires scripts and tools to exercise, which takes longer than the available time. However, if you have tools prepared from previous testing and this feature is sufficiently similar that you can quickly modify them, you can run initial performance and reliability testing (see Chapter 12, Load Testing, for more details). Again, this isn’t a priority compared to the usability and functional testing elements.

As you go through these types of testing, note down ideas and areas that you should cover in the complete test plan. Exploratory testing is a time to try out some ideas, but also to identify what you want to spend more time on later. You won’t be able to cover everything initially, so record what you have missed so that you can return to it.

This is also your chance to uncover trends within the bugs. Defects aren’t randomly scattered through a product; they are grouped. So, check what parts of this feature or product are particularly suffering from issues. Adapt your test plans to add extra detail in those areas, to find further problems. This lets your testing be reactive to the feature, to give the best chance of finding bugs. See Chapter 5, Black-Box Functional Testing, for more on reactive testing.

By the end of this testing, you should have a very good idea of how this feature works and have ensured it doesn’t have any major issues. Usually, that will prepare you for more detailed testing to follow, but sometimes, it is all that is possible, as the next section explains.