Using curiosity in testing

The approach to take throughout exploratory testing is that of maximum curiosity. Constantly ask yourself, “I wonder what happens if I do that?” All the questions you can think of are valuable while testing; testers are there to ask the right questions.

All testing requires that same curiosity, but you can see it most clearly in exploratory testing, where there is no test plan, and you only have your curiosity to work from.

All curiosity is not equal, however, and exploratory testing is an area that benefits massively from experience. With only a limited time, you must aim directly for the weak spots in your system to avoid wasting time testing mature code that has already been well covered in previous cycles or with automated testing.

One area of potential weakness is the new change that hasn’t had any test coverage before. But what existing features should you combine with these new ones to find problems? This is where your experience comes into play. What are the weaknesses of your system? Where have bugs been seen before?

The International Software Testing Qualifications Board (ISTQB) is an organization dedicated to improving the software testing practice, which runs certification schemes for practitioners. They note that bugs are not evenly spread throughout code but tend to cluster in certain areas.

Examples of classic areas for problems that affect many systems are as follows:

• Running on low specification machines with limited memory or CPU
• Running on low-resolution screens or small window sizes
• Behavior after the upgrade when you transition to using the new feature
• Problems around boundaries and edge conditions
• Problems with times and time zones
• Problems with text inputs (blank, too long, Unicode characters, SQL injection, and more)
• Backup and restore
• Behavior under poor network conditions

In addition to the preceding problems, track all the weaknesses that affect your particular application or web page. Create a document that lists the areas of weakness that you’ve hit in the past, and keep it updated as you discover new issues.

This technique is known as error guessing, as described in the ISTQB syllabus. It stresses that the tester’s experience is needed to anticipate where the errors might fall. Often, you find the best bugs when you go off the test plan. This is the corollary to the ISTQB testing principle about the weedkiller paradox – the more you test in one particular way, the fewer bugs you find, analogous to using a weedkiller that gradually kills fewer weeds. Only those less affected by it thrive. Tests that you have run many times before are unlikely to find an issue this time around, and in addition, the idea behind them may be well known to the development and test teams, so the developers will remember to handle that case. You’re more likely to find bugs with new ideas for tests and in new combinations of functionality that haven’t been previously considered.

Because of the dependence on experience for exploratory testing, it is best carried out by senior testers. Junior testers can cover other parts of the testing process, such as implementing test plans designed by others or regression testing. For exploratory testing, make sure experienced team members lead the way. If you are a test manager, you can make that decision; if you are a tester who has been given the task, let others know if there is someone more suitable. Experience can be domain-specific, of course. While you may be highly experienced in one area, perhaps someone else is better placed to do exploratory testing for this particular feature.

Experience is also important when it comes to choosing what to check. While some errors are visible on the user interface, others may be subtle, such as warnings in the logs or database fields being written incorrectly. As well as using your curiosity when deciding what to do, ensure you scour the system for issues and items you can check.

Curiosity is vital throughout the test process, not just during exploratory testing. There is no hard divide between ad hoc testing and that which follows a plan. Instead, it’s a spectrum. While exploratory testing has little structure, lists of likely weaknesses such as the preceding one can also help to guide it. So, even exploratory testing can have some documentation.

Functional testing can be descriptive – for example, a test such as “Upload a .jpeg image," which means that each tester will choose a different image to use. That reduces the reproducibility of the test while broadening coverage. Alternatively, tests can be prescriptive, describing exactly what to do; for example, “Upload image test1.jpeg." Even within these tests, the environment can change, the state of the system may be different, or you might run tasks in an unusual order. Always look out for new ways to test and new things to try.

Testing is an arms race of creativity, with testers trying to find new ways to break products and developers inadvertently adding new problems and issues. This is the fun and the skill of testing, so go crazy and enjoy yourself! When testing a new feature, you are the first to do something no one has done before. You are the vanguard of the army, charging into battle; you are the pioneer heading into an unknown land. So, use your experience, prepare for surprises, and keep your eyes open for the unexpected.