Scope-based Compiler Differential Testing

Compilers are among the most critical components in the software development. Obviously, their correctness is very important, yet they are among the most complex software systems. The traditional grammar-based compiler random testing measures have two shortcomings. Firstly, the technique generating test programs for one programming language is difficult to migrate to another. The second one is that traditional grammar-based technique haven’t optimized the relation of identifier definition and use yet, causing the undefined identifiers problems or the low quality of the generated test programs. To address these problems, we propose a scope-based compiler testing method ScopeGen in this paper. To generate runnable and diverse test programs, ScopeGen supports two types of identifier strategies based on the scope information, one is scope distance based and the other is global optimization based. These identifier strategies guide the definition and use of identifiers and balance the distribution of identifiers in different scopes. Benefiting from the public grammar dataset Grammar-v4, ScopeGen can be easily migrated to various programming languages. We implement a program generator and generate grammatically correct and runnable test programs for C, Java and Python. Next, we conduct differential testing to identify various bugs in compilers by comparing the output of different compilers. The experimental evaluation of 9 compilers (gcc, clang, icc, icx, Ark, Javac, CPython, Pypy and Codon) shows that ScopeGen outperforms the two state-of-the-art methods (i.e., Csmith and YARPGen) improving more than 69% in inconsistency finding ability. By running ScopeGen we have reported 114 bugs for 4 compilers, 84 of which were confirmed.