Can LLMs Generate Higher Quality Code Than Humans? An Empirical Study

Large Language Models are being extensively used for AI-assisted programming and code generation. The challenge is to ensure that the generated code is not only functionally correct but also safe, reliable and trustworthy. In this direction, we conduct a comprehensive empirical analysis of AI-generated code to assess whether large language models (LLMs) can produce correct and higher-quality code than humans. We evaluate the code quality of 984 code samples generated by GPT-3.5-Turbo and GPT-4 using various prompt types (simple, instructional, and enhanced) against input queries from the HumanEval dataset. We also enhance the HumanEval benchmark by calculating code quality metrics for the human-written code it contains. Code quality metrics are calculated using established tools like Radon, Bandit, Pylint, and Complexipy, with human-written code serving as a baseline for comparison. To quantify performance, we employ the TOPSIS method to rank the models and human code by their proximity to ideal and anti-ideal code quality metrics. Our results demonstrate that GPT-4, when used with advanced prompts, produces code closest to the ideal solution, outperforming human-written code in several key metrics. Our work provides evidence that LLMs, when properly guided, can surpass human developers in generating high-quality code. Our code and datasets are available online.