Exploring Regression of Data Race Detection Tools Using DataRaceBench

DataRaceBench is an OpenMP benchmark suite designed to systematically and quantitatively evaluate data race detection tools. It has been used by several research and development groups to measure the quality of their tools. In this paper we explore how to evaluate the regression of data race detection tools in the presence of observed tool errors. We define how to generating consistent, reproducible, and comparable evaluation results and a detailed evaluation process with a set of configuration and execution rules. We also outline differences in the evaluation of dynamic and static data race detection tools. In addition to the evaluation results, we explore and suggest different ways to process and present the data, with a focus on tool errors. Using DataRaceBench we show an accuracy regression for several popular data race detection tools in recent release cycles.