Machine learning differential testing

Differential testing attempts to check two versions of a piece of software, considered as base and test versions, on the same input and then compare the outputs. This process helps us identify whether the outputs are the same and identify unexpected differences (Gulzar et al., 2019; Figure 8.3):

Figure 8.3 – Simplified flowchart of differential testing as a process to test the outputs of two implementations of the same process on the same data

In differential testing, the base version is already verified and considered the approved version, while the test version needs to be checked in comparison with the base version in producing the correct output. In differential testing, we can also aim to assess whether the observed differences between the outputs of the base and test versions are expected or can be explained.

In machine learning modeling, we can also benefit from differential testing when comparing...