Specification based testing with QuickCheck

QuickCheck is a tool which tests software against a formal specification, reporting discrepancies as minimal failing examples. QuickCheck uses properties specified by the developer both to generate test cases, and to identify failing tests. A property such as ∀xs : list(int()). reverse(reverse(xs)) = xs is tested by generating random lists of integers, binding them to the variable xs, then evaluating the boolean expression under the quantifier and reporting a failure if the value is false. If a failing test case is found, QuickCheck “shrinks” it by searching for smaller, but similar test cases that also fail, terminating with a minimal example that cannot be shrunk further. In this example, if the developer accidentally wrote ∀xs : list(int()). reverse(xs) = xs instead, then QuickCheck would report the list [0, 1] as the minimal failing case, containing as few list elements as possible, with the smallest absolute values possible. The approach is very practical: QuickCheck is implemented just a library in a host programming language; it needs only to execute the code under test, so requires no tools other than a compiler (in particular, no static analysis); the shrinking process “extracts the signal from the noise” of random testing, and usually results in very easy-to-debug failures. First developed in Haskell by Koen Claessen and myself, QuickCheck has been emulated in many programming languages, and in 2006 I founded QuviQ to develop and market an Erlang version. Of course, customers' code is much more complex than the simple reverse function above, and requires much more complex properties to test it. The challenge in applying QuickCheck to real code is in finding ways to formulate properties that are simple enough for people to use easily, concise enough to make property-based testing cost-effective, and avoid the trap of replicating the mistakes of the implementation in the specification. To this end we have extended QuickCheck with state machine formalisms, and standardized serializability properties that can expose harmful race conditions in concurrent code. I will present examples using these formalisms, and discuss our experiences of applying property-based testing in the telecoms, automotive, and distributed database industries.