When in Doubt Throw It Out: Building on Confident Learning for Vulnerability Detection

[Context:] Confident learning’s intuition is that a good model can be used to identify mislabelled data. By swapping mislabeled samples that are not confidently predicted, the performance of model can be further improved. [Problem:] Unfortunately, vulnerability detectors are generally under-performing models and confidence learning would conclude that the bulk of the dataset is mislabelled. [New Idea:] We extend confidence learning by identifying a type of training samples that appear in presence of under-performing models: confusing samples. [Emerging Result:] We analyze the formal constraints for confusing samples and perform preliminary experiments that show that the model’s performance is effectively improved after deleting confusing samples entirely from the training set.