Writing tests for regression

When developers try to convince managers that testing is something that is necessary to their project, regression is often a card that is drawn. Managers may claim the tests can be run in a build server to make sure functionality is continuously verified. While this is true, it is more of a side effect, unless tests are written for this specific reason.

A good regression test states something about the functionality that is always true. It should not be so much dependent on the implementation but on the specification of the functionality.

I was once working with a client on a system that was fairly complex. It was an order process that was divided into several steps to minimize the complexity for the end user, but still, there were hundreds of business rules implemented in the backend.

I was working alone one evening in the office when a business analyst came rushing in, claiming that I needed to take the website down. After some querying, he told me that the discount logic for students was wrong.

With the business analyst standing over my shoulder, I went into my test suite and found the following regression test:

PriceCalc_Should_Discount_Students_With_Child_Under_16_y o()

The test was turning into green as I ran it, and I asked him for his test data. It turned out he was using his own personal information as test data and had a daughter that had recently turned 16.

One peculiar observation about bugs is that they have a tendency to come back, unless carefully observed. This is why it's always best to write a regression on finding a bug to make sure it doesn't reappear. Personally, I write these tests to verify the claim of the bug and then use the test to show me when the bug is fixed.

Executable specifications

Written tests inform from an outside perspective how the system is behaving. This is a very powerful concept that, if enriched, will lead to tests as specifications. Looking at the tests will tell you how the system works.

Having a large test suite could easily replace the requirements and specifications of the system, as the test suite verifies the stated functionality every time tests are run. The documented specifications and requirements become outdated after the first change.

I was once consulting for a client that was going to sell gym memberships online. The implementation itself was not that hard: gather customer information and store it in a Customer Relationship Management (CRM) system. The credit card payment was hosted by an external payment provider and integrated with some basic HTTP redirects.

However, the client was insistent on having very complex price logic to a degree where it was impossible for one person to understand why a membership had been assigned a target price.

In order to implement this, I started from the requirements and wrote them all as tests. I felt confident that my test suite covered the whole problem and would implement the system in such a way that my test would turn green.

Turning over the solution to the client for User Acceptance Testing (UAT), I got back 10 scenarios where the client claimed the membership had the wrong price.

Still confident in my method of implementation, I chose to implement all failing scenarios as tests. It proved that the code was not wrong, but the logic was so complex that the client couldn't verify it in UAT.

After some iteration with this, the client finally gave up acceptance testing and had us release it to production. As a consultant, I should have advised my client to simplify their price logic.

What tests are trying to achieve as specifications is to have executable specifications written in a natural language that can verify the following:

• The code is implemented as specified
• The code keeps fulfilling the specification (regression)

While specifications are being written in natural language, it is possible for programmers and business analysts to have a common workspace on how the system is supposed to work.

The following example shows the specifications written in a natural language:

Feature: Authentication

Scenario: Entering correct login information makes user authenticated
  Given a fresh browser session at http://mikaellundin.name/login
  When entering 'mikaellundin' as username
  And entering 'hello fsharp' as password
  Then browser should redirect to http://mikaellundin.name/profile

The specification is written in a Domain Specific Language (DSL) called Gherkin. Each line has code connected to it that will execute when the specification itself is executed to verify that the requirement is fulfilled.