Eradicating flaky tests
Non-deterministic or flaky tests are tests that sometimes pass and sometimes fail with the same code (Martin Fowler, 2011). Flaky tests can destroy the trust in your test suite. This can lead to teams just ignoring red test results, or developers deactivating tests, thereby reducing the test coverage and reliability of the suite.
There are lots of reasons for flaky tests. Often, they are due to a lack of isolation. Many tests run in the same process on a machine – so each test must find and leave a clean state of the system. Another common reason is asynchronous behavior. Testing asynchronous code has its challenges as you never know which order the asynchronous tasks are executed in. Other reasons may include resource leaks or calls to remote resources.
There are different ways to deal with flaky tests:
- Retry failing tests: Some frameworks allow you to retry failing tests. Sometimes, you can even configure a higher level of isolation. If...
