Can LLMs Generate Green Code - A Comprehensive Study Through LeetCode

This paper presents a comprehensive analysis of code generated by Large Language Models (LLMs) in terms of energy consumption and speed, moving beyond traditional accuracy metrics. We evaluate eight state-of-the-art models, including ChatGPT-3.5, ChatGPT-4o, CodeGemma:7b, WizardCoder:33b, Llama3:8b, Phind-CodeLlama:34b-v2, Nous-Hermes2:10.7b, and Mistral:7b, using RuntimeRatio and EnergyRatio metrics on LeetCode¹ questions. To evaluate LLMs’ understanding on energy efficient concepts, we propose a three-step prompting methodology to test models on coding with runtime and energy efficiency in mind. In an overwhelming majority of cases, LLMs perform worse than the human solution in both code runtime and energy consumption. In cases where models do perform well, it is often because they are aware of the question online, and produce responses similar or identical to famous solutions. Our comparative analysis of the efficient prompt and chain of thought prompt methods to the base prompt emphasize that models are inconsistent with producing quicker and more efficient code when directly asked to assemble energy efficient code. Our findings highlight that LLMs currently lack understanding in energy efficiency in terms of code generation, pushing a need for efforts to add green computing ideas and energy efficiency for future training and fine-tuning.