Architecture and Execution Flow

CMakeTest is built around the concept of an Execution Unit. A unit is an object representing a test or test section and may contain other subunits that represent subsections. An execution unit contains all of the information needed to run the test including the containing file, the test ID, and whether its EXPECTFAIL option is set. The object system is provided by CMakePP and the class that represents execution units is CTExecutionUnit.

Execution Stages

CMakeTest runs in two stages: the configure stage and the test stage. The test stage is executed by a ctest command in the directory that tests are built in.

Configure Stage

During the configure stage, CMakeTest scans directories that were added via ct_add_dir and generates a bootstrap CTest test file for each CMake file in each directory. The bootstrap file is generated from a template, which is explained in Bootstrap Templates.

Test Stage

The test stage is the phase in which the majority of CMakeTest code is executed. The generated bootstrap files handle the setup of the CMakeTest runtime. From there the bootstrap sequence begins the test discovery and execution procedure.

Test Discovery

Tests are discovered by include()-ing the file under test, which will execute all top-level ct_add_test calls. These calls instantiate one CTExecutionUnit object each and add it to a CMAKETEST_TEST_INSTANCES CMakePP global list. This initial discovery procedure does not discover subsections of tests.

Test Execution

Once all tests are discovered, they are then executed via ct_exec_tests. This function first registers the global exception handler using ct_register_exception_handler and then loops over the execution units stored in CMAKETEST_TEST_INSTANCES, calling the associated execute method.

The execute() method is the meat of the test execution. It begins by setting some needed globals, and then checks if the test is EXPECTFAIL. If so, the test is executed via ct_expectfail_subprocess. Otherwise, the test is executed via calling the function directly using CMakePP’s cpp_call_fxn() command.

During the test execution, subsections are discovered but not executed. The discovery happens in a similar vein as the test discovery, i.e. ct_add_section instantiates a new execution unit and adds it to the currently-executing unit’s children map.

Once the test has finished executing, a pass line is printed if no errors were detected. The test’s subsections are then executed via the unit’s exec_sections method.

Error Catching and Recovery

CMake does not have any concept of error recovery, any message(FATAL_ERROR ...) call will immediately terminate the interpreter. CMakeTest must be able to catch these errors, log the error and label the test as failing, and continue executing tests. Additionally, CMakePP includes a form of exceptions that also need to be caught before they result in interpreter termination. The biggest problem with these exceptions is that they cannot unwind the stack, so once the exception handler returns the execution continues unabated from the throw point.

CMakeTest handles these problems using a few somewhat-“hacky” tricks. The first of these tricks overrides the builtin message() function with a custom one (cmake_test/overrides.message). This override converts all FATAL_ERROR messages to CMakePP exceptions. This allows all vanilla CMake code running under tests to error without immediately terminating the interpreter, including code in external modules.

The second trick exists in the exception handler registered by ct_register_exception_handler. The handler is registered for all exceptions, which means it also handles the generic exceptions generated by our message() override. When the handler catches an exception it checks the execution unit that is currently running (tracked by a global), sets the has_printed and has_executed attributes to true, and then calls ct_exec_tests again.

This may seem counter-intuitive, but by restarting the execution and by checking the previously-set variables we can selectively skip tests that have already ran and only run those that have not ran. This works around the inability to unwind the stack. Once the ct_exec_tests() call returns we call ct_exit to terminate the interpreter, which prevents the code up the stack from continuing.