Using exploratory testing by necessity

While normally used at the start of testing large features, exploratory testing is also vital when you don’t have time for anything else. As we’ve seen, exploratory testing combines test design, execution, and interpretation into a single step and is highly efficient at the cost of having less planning and documentation. In the hands of an experienced tester, it can provide broad coverage quickly and rapidly gain confidence in a change.

This is useful toward the end of a project when there are minor changes to fix critical bugs. You need to check those changes, but you must also perform regression testing to ensure that no other errors have been introduced. Regression testing involves carrying out tests on existing functionality to make sure it still works and hasn’t been broken. Exploratory testing mainly focuses on new behavior, but it can also check your product’s main functions.

Real-world example – Fixing XML

At the end of a long, waterfall development cycle that lasted over 6 months, we were almost ready to release a new version of our product. We found one last blocking bug that required a new build, so the development team made that change and provided us with the release candidate code. Unfortunately, they hadn’t only fixed the blocking bug. While in that area, the developer had noticed that our XML formatting was wrong and had taken the opportunity to fix it.

The developer was right. The formatting was incorrect, but he hadn’t realized that the system that read that data relied on that incorrectness. By fixing the formatting, the other system could no longer read the message. That critical bug was introduced at the end of the release cycle, but we quickly found it through exploratory testing. It delayed the whole project since everything had to stop to wait for the fix, but it didn’t take long to roll back the change.

When pressed for time, exploratory testing is the fastest way to get broad coverage of a change. You can rapidly gain confidence that a feature is working but beware of overconfidence if these are the only tests you have run. There can still be blocking issues even after exploratory testing has passed. Non-functional tests are often poorly covered by exploratory testing: does the feature work on all web browsers and operating system versions? Does it work with a high load? Does it handle poor network conditions? Even if the feature works well in one tester’s environment, real-world situations may hit problems.

So, be aware that exploratory testing is not comprehensive. Rather than putting more effort into this form of testing, use it to plan out exhausting test plans that will be documented and automated. Even when you have more time, you should always limit how long you spend on this form of testing.