A Framework of Vulnerable Code Dataset Generation by Open-Source Injection

Evaluation benchmark plays an important role in the design of defect detection algorithms and tools. Especially with the development of deep learning techniques, code defect detection models based on deep neural network requires a large number of training and testing cases. Existing test cases are far from meeting the requirements of new algorithm design and verification. On the one hand, the number of test cases designed manually or collected from open source projects is small. On the other hand, the test cases generated automatically according to rules have similar pattern, high redundancy and simple structure. This paper proposes an algorithm of code defect injection and test case generation based on open source projects. The basic idea is to find reaching definitions in open source projects, and modify the source code according to the analysis results, so as to generate defect dataset with a large number of test cases that have similar feature to open source codes. This paper selects 8 open source projects to verify the proposed method and generates more than 6,000 null pointer dereference test cases in total. We use existing tools to evaluate the injected test cases and the results show that the proposed method can generate a large number of high-quality test cases.