Write me this Code: An Analysis of ChatGPT Quality for Producing Source Code

Developers nowadays are increasingly turning to large language models (LLMs) like ChatGPT to assist them with coding tasks, inspired by the promise of efficiency and the advanced capabilities they offer. However, this raises important questions about the ease of integration and the safety of incorporating these tools into the development process. To investigate these questions, this paper examines a set of ChatGPT conversations. Upon annotating the conversations according to the intent of the developer, we focus on two critical aspects: firstly, the ease with which developers can produce suitable source code using ChatGPT, and, secondly, the quality aspects of the generated source code, determined by the compliance to standards and best practices. We research both the quality of the generated code itself and its impact on the project of the developer. Our results indicate that ChatGPT can be a useful tool for software development when used with discretion.CCS CONCEPTS• Software and its engineering → Reusability; Open source model; Software defect analysis; Software libraries and repositories.