Exploring the TestSuite framework

In the previous sections, we learned how regression tests were performed in LLVM. More specifically, we looked at the ShTest testing format (recalling the config.test_format = lit.formats.ShTest(…) line), which basically runs end-to-end tests in a shell script fashion. The ShTest format provides more flexibility when it comes to validating results since it can use the FileCheck tool we introduced in the previous section, for example.

This section is going to introduce another kind of testing format: TestSuite. The TestSuite format is part of the llvm-test-suite project – a collection of test suites and benchmarks created for testing and benchmarking LLVM. Similar to ShTest, this LIT format is also designed to run end-to-end tests. However, TestSuite aims to make developers' lives easier when they want to integrate existing executable-based test suites or benchmark codebases. For example, if you want to use the famous SPEC benchmark...