An Investigation of Quality Issues in Vulnerability Detection Datasets

Vulnerability detection is a crucial yet challenging task in ensuring software security, and deep learning (DL) has made significant progress in automating this process. However, a DL model requires a massive amount of labeled data (vulnerable and secure source code) to effectively distinguish between the two. Many datasets have been created for this purpose but they suffer from several issues that can lead to low detection accuracy of DL models. In this paper, we define three critical issues (data imbalance, low vulnerability coverage, and biased vulnerability distribution) and three secondary issues (errors in source code, mislabeling, and noisy historical data) that can affect the model performance. We also conduct a study of 14 papers along with 54 datasets for vulnerability detection to confirm these defined issues. Furthermore, we discuss good practices for using existing datasets and creating new ones to improve the quality of data available for automated vulnerability detection. This paper aims to raise awareness of the importance of data quality in vulnerability detection and provide proper guidelines for researchers and practitioners working in this area.