A failure injection tool for testing soft and hard real time protocols

We present an approach to testing implementations of real-time and fault-tolerant protocols. In our object-oriented tool, implementations are subject to real interactions with the environment, and to the same failure scenarios they are designed to tolerate. A centralized simulator executes all the tasks of the distributed system on a single address space. Testing is nonintrusive; the timing of events is exactly the same in real and simulated executions. Experiments can be exactly reproduced any number of times. Our tool helped us find several flaws in the implementation of a fault-tolerant clock synchronization protocol, and get detailed performance measurements that would have been extremely difficult to get in a real distributed system.