An Automated (F) LTL Test Oracle for Testing with Requirements

Especially for safety-critical systems, we might want to express desired system properties in precise formal requirements. Verifying that a system then adheres to these requirements is obviously quite crucial. Complementing related methods like model checking or runtime monitors, for testing and most importantly debugging recognized problems, we would certainly be interested in automated test oracles for deriving program spectra. Such program spectra containing the details of the executions and the oracle's verdicts have been shown to be an effective reasoning basis for debugging purposes. In this paper, we show how to automatically derive such a test oracle via a special satisfiability encoding. In particular, we instantiate a dedicated SAT problem in conjunctive normal form directly from the requirements and a test case's execution data. As our experiments illustrate, our approach shows attractive performance and can be fully automated.