Testing as a practice

Before diving into why we need test automation, we should consider what it really is. The practice is still quite new and there is some confusion surrounding it, leading to developers testing the wrong thing and managers not knowing what to expect.

Black or white box testing

Testing practices are often split into black or white box tests. The difference is distinguished by how much we know about the system we're testing. If all we know about the system is what we can see from the outside and all we can do with it is build outward interfaces, then the method of testing is considered a black box test.

On the other hand, if our testing knows about the inward functions of the system and is able to trigger events or set values within it, then this testing is referred to as white box testing.

These are two slightly different viewpoints to consider when testing. When performing test automation, we need to examine both black and white box testing, where white box testing is closer to the implementation and black box testing is often leaned toward based on a user requirement's abstraction level.

Manual testing

Manual testing is a practice used to investigate a product and determine its quality. This is done by a person called a tester and is performed by executing the program or using a tool to examine it. The testing will validate that the product meets its requirements and determine if the system is usable, but most importantly, it will validate that the product solves the problem it was created for.

The following image shows how testing fits into the normal flow of software development:

The result of manual testing is a set of issues that gets reported to the development team. Some of these issues are labeled as bugs, defects, or just changes. The tester will rate them based on priority (blocker, critical, high, medium, or low) and incorporate them into the development process.

The term manual testing is usually just called testing, but to avoid confusion, I will refer to testing done by a tester as manual testing and testing that is executed by a computer as test automation.

Test automation

Test automation is a practice used to create checks that will verify the correctness of a product. These checks are written in code or created in a tool, which will then be responsible for carrying out the test. The nature of these checks is that they are based on the requirements and are reproducible through the automation.

The following image shows how test automation doesn't require a tester or need for reports and issue tracking:

Most commonly, test automation is performed by the development team and is an integrated part of the software development process. It doesn't replace the tester but puts an extra layer of quality assurance between the team and tester, leading to fewer issues reported by the tester.

The best kind of testing is that which requires little effort. The code is reviewed by the computer when compiling the program, verifying that it's possible to turn the code into machine instructions. For a statically typed language, this can be seen as the first line of testing, like a spell check.

Once the code is compiled, the programmer understands that the code will be executed. It will not necessarily do what it's supposed to do, but it's guaranteed to execute, which is not always the case if interpreted at runtime.

The following table shows the layers of testing and what they verify:

Test activity	Input	Verifies
Compiling	Source code	Syntax correctness
Style check	Source code	Code style
Static analysis	Source code / compiled assembly	Code correctness
Unit testing	Compiled assembly	Code correctness
Integration testing	Compiled assembly	Code behavior
System testing	Release version	Product behavior

Style check

A style check on the code will ensure it is properly formatted and enforces conventions such as the name standard, indenting, comments, and so on. This is very valuable in a team setting to increase readability of the code and maximize code sharing, as all developers use the same coding style. The result is higher quality and less friction, leading to fewer bugs and faster development.

For F#, there is a style-checking tool called FSharpLint, which is available through the NuGet package manager and can be used to check your code against style conventions.

Static analysis

Static code analysis can be used to avoid unnecessary mistakes, including unintended circle references or badly implemented patterns, such as poor implementation IDisposable. It helps in avoiding problems that inexperienced developers would have with garbage collection and threading.

Tip

There are no good static analysis tools for F# as of this writing. In C#, one could use Visual Studio Code Analysis, previously known as FxCop, for static analysis.

Unit testing

Unit tests are written at the same time as the code, before or after. They verify that the code produces the intended result. This is a form of white box testing that seeks to reduce the number of unintended defects that come out of development. If the unit testing is thorough, the code will do what the programmer intended.

Here's an example unit test:

open NUnit.Framework
open FsUnit

[<Test>]
let ``should return 3 from adding 1 and 2`` () =
    Calculator.add 1 2 |> should equal 3

Integration testing

An integration test is a test written by the programmer to verify his or her code's integration with other systems or parts of the system, such as databases and web services. The purpose of this testing is to find problems and side effects that only appear in the integration with those other systems. If integration testing is thorough, it will help with the stability of the system.

Here's an example integration test:

open NUnit.Framework
open FsUnit

[<Test>]
let ``should store new user to data storage`` =
    // setup
    let newCustomer = { name = "Mikael Lundin"; address = "Drottninggatan 82 Stockholm" }

    // test, storing new customer to database
    let customerID = CustomerRepository.save newCustomer

    // assert
    let dbCustomer = CustomerRepository.get customerID
    dbCustomer |> should equal newCustomer

System testing

System testing is a form of black box testing that is performed in order to validate that the system requirements, both functional and nonfunctional, are fulfilled. System testing is a very broad term and is more often pushed to manual testing than it is automated. Executable specifications is one area where system testing automation excels.