Testing Constraint Checking Implementations via Principled Metamorphic Transformations

Constraint checking techniques are being widely used for ensuring the consistency of software artifacts during their development and evolution (e.g., detecting inconsistency in an application's running contexts or identifying rule violation in the code being developed). Typically, consistency constraints are formulated and checked upon the changes of software artifacts under checking. When any constraint is violated, an inconsistency is said to occur and then follow-up actions can be taken to remedy the problem. Currently, various constraint checking techniques have been proposed and implemented with sophisticated mechanisms for higher efficiency and scalability. However, these implementations could be far from being well tested due to their oracle problems, i.e., hardly able to tell what the checking result should be, given any software artifacts and their consistency constraints to check. In this paper, we leverage metamorphic testing and propose a family of metamorphic relations catered for testing constraint checking implementations. We dedicatedly design these relations by following the sensitivity principle via a fine-granularity control and the diversity principle via input-oriented transformations. Our experiments reported promising results (disclosing 80 % mutation bugs and five real bugs) without the need of any manual labeling.