Artificially Insecure: Examining GitHub Copilot’s AI-based Vulnerability Prevention System

AI-assisted programming has quickly become standard practice in software development life cycles. GitHub Copilot is a popular AI pair-programmer used by developers and enterprises. The aim of these tools is to help developers write better code faster. However, recent research has shown that AI-assisted programming tools often produce code that contains at least one vulnerability. The creators of these tools are working to prevent vulnerable coding patterns from being generated. For example, GitHub has implemented new security features that prevent vulnerable coding patterns from being generated. However, subsequent research has shown that issues persist, and continued work is needed. In this paper we evaluate Copilot’s current performance generating code in Python, JavaScript and C, when presented with Common Weakness Enumerators (CWE) scenarios from the LLMSecEval dataset. These results and their comparison to past studies shed new light on the degree secureness of code generated by Copilot has improved and provides a fresh benchmark for future studies. Our results show that despite Copilot producing 97.03% functionally correct code snippets, 30.6% contained at least one vulnerability. In addition, the results of this experiment found that less secure code is generated when working in JavaScript than snippets produced in Python and C. Consequently, developers should continue to critically evaluate code generated by Copilot.