Making mistakes while writing a DBI framework is particularly easy and often fatal. Executing code generated at runtime is a dangerous game and close attention should be paid to the patching engine. That is why we use two different testing methods.
Functional Test Suite
QBDI has a small functional test suite implemented using Catch2
which verifies the essential functions of QBDI. This test suite is automatically built and made
available in the
test/ build subdirectory:
$ ./test/QBDITest =============================================================================== All tests passed (110977 assertions in 106 test cases)
The validator is a generic test system allowing to compare a normal execution with an instrumented execution. This way a large number of programs can be used as a test suite to ensure that QBDI does not alter their normal functions. It is only compatible with Linux and macOS at the moment.
The idea is to pilot a debugging session using an instrumented execution of the same program. These two instances share the same environment and arguments and originated from the same fork thus have the same memory layout. This allows to compare the program state at each step and verify that both execution take the same path and give the same result. The validator is not only capable of determining if two executions differ but also capable of identifying where they diverged. It thus double down as a debugging tool.
There is, however, a few caveats to this approach. First, the two instances of the program will
compete for resources. This means running
sha1sum test.txt will work because the two instances
can read the same file at the same time, but removing a directory
rmdir testdir/ will always fail because
only one instance will be able to delete the directory. Second, the dynamic memory allocations will
not match on the two instances. This is because the instrumented instance is running the whole QBDI
framework and validator instrumentation which is making extra allocations in between the original
program allocations. A partial mitigation was nevertheless implemented which tracks allocations on both
sides and compute an allowed offset for specific memory address ranges.
One of the essential concepts of the validator are error cascades. They establish a probable causality chain between errors and allow to backtrack from the point where the execution crashed or diverged to the probable cause. Indeed, an error might only cause problems thousands of instructions later.
The validator uses dynamic library injection to take control of a program startup. Options are communicated using environment variables. The following options are available:
Stat: Only display execution statistics. This is the default.
Summary: Display execution statistics and error cascade summaries.
Detail: Display execution statistics and complete error cascades.
Full: Display full execution trace, execution statistics and complete error cascades.
Specify a file name where instruction coverage statistics will be written out.
The validator uses
LD_PRELOAD for injection under Linux and an example command line would be:
$ LD_PRELOAD=./tools/validator/libvalidator.so VALIDATOR_VERBOSITY=Detail VALIDATOR_COVERAGE=coverage.txt ls
The validator uses
DYLD_INSERT_LIBRARIES for injection under macOS and an example command line would be:
$ DYLD_INSERT_LIBRARIES=./tools/validator/libvalidator.dyld VALIDATOR_VERBOSITY=Detail VALIDATOR_COVERAGE=coverage.txt ./ls
Please note that, under macOS, SIP prevents from debugging system binaries. A workaround is to copy the target binary in a local directory. Also, for the moment, the validator requires root to obtain debugging rights.
The validation runner is an automation system to run a series of validation task specified in a configuration file and aggregate the results. Those results are stored in a database which enables to perform historical comparison between validation runs and warns in case of an anomaly.