Poster: Identifying Security Issues in Software Development: Are Keywords Enough?

Identifying security issues before attackers do has become a critical concern for software development teams and software users. One approach to identifying security issues in software development artifacts is to use lists of security-related keywords to build classifiers for detecting security issues. However, generic keyword lists may miss project-specific vocabulary. The goal of this research is to support researchers and practitioners in identifying security issues in software development project artifacts by defining and evaluating a systematic scheme for identifying project-specific security vocabularies that can be used for keyword-based classification. We sampled and manually classified 5400 messages from the Apache Derby, Apache Camel, and Dolibarr projects to form an oracle. In addition, we collected each project's publicly disclosed vulnerability data from the CVE and mapped them to the project's dataset to create a CVE-labelled data. We extracted project-specific vocabulary from each project and built classifiers to predict security-related issues in both the oracle and CVE dataset. In our data, we found that the vocabularies of each project included project-specific terms in addition to generic security keywords. Classifiers based on the project-specific security vocabularies increased recall performance by at least double (at varying costs to precision) compared with the previous published keyword lists we evaluated.