High-Precision Evaluation of Both Static and Dynamic Tools using DataRaceBench

DataRaceBench (DRB) is a dedicated benchmark suite to evaluate tools aimed to find data race bugs in OpenMP programs. Using microbenchmarks with or without data races, DRB is able to generate standard quality metrics and provide systematical and quantitative assessments of data race detection tools. In this paper, we present a new version of DRB with several improvements. First, we design a novel approach to enable high-precision checking of tool results. The approach relies on a format to accurately encode data race ground truth including variables, read/write types, and source file location information. The test harness of DRB has also been improved to support static data race detection tools. Finally, an enhanced code similarity analysis is developed to consider code region details and cover more regions. Our experiments show that the improved DRB generates more accurate reports and exposes more limitations of both static and dynamic data race detection tools. The enhanced similarity analysis also is able to guide us to investigate similar code regions in DataRaceBench in more detail.