SPEcBCFuzz: Fuzzing LTL Solvers with Boundary Conditions

LTL solvers check the satisfiability of Linear-time Temporal Logic (LTL) formulas and are widely used for verifying and testing critical software systems. Thus, potential bugs in the solvers' implemen-tations can have a significant impact. We present S
PEC
BCF
UZZ
, a fuzzing method for finding bugs in LTL solvers, that is guided by boundary conditions (BCs), corner cases whose (un)satisfiability depends on rare traces. S
PEC
BCF
UZZ
implements a search-based algorithm that fuzzes LTL formulas giving relevance to BCs. It inte-grates syntactic and semantic similarity metrics to explore the vicinity of the seeded formulas with BCs. We evaluate S
PEC
BCF
UZZ
on 21 different configurations (including the latest and past releases) of four mature and state-of-the-art LTL solvers (NuSMV, Black, Aalta, and PLTL) that implement a diverse set of satisfiability algorithms. S
PEC
BCF
UZZ
produces 368,716 bug-triggering formulas, detecting bugs in 18 out of the 21 solvers' configurations we study. Over-all, S
PEC
BCF
UZZ
reveals: soundness issues (wrong answers given by a solver) in Aalta and PLTL; crashes, e.g., segmentation faults, in NuSMV, Black and Aalta; flaky behaviors (different responses across re-runs of the solver on the same formula) in NuSMV and Aalta; performance bugs (large time performance degradation between successive versions of the solver on the same formula) in Black, Aalta and PLTL; and no bug in NuSMV BDD (all versions), suggesting that the latter is currently the most robust solver.