Commit Message Generation via ChatGPT: How Far are We?

Commit messages concisely describe code changes in natural language and are important for software maintenance. Various automatic commit message generation approaches have been proposed, such as retrieval-based, learning-based, and hybrid approaches. Recently, large language models have shown impressive performance in many natural language processing tasks. Among them, ChatGPT is the most popular one and has attracted wide attention from the software engineering community. ChatGPT demonstrates the ability of in-context learning (ICL), which allows ChatGPT to perform downstream tasks by learning from just a few demonstrations without explicit model tuning. However, it remains unclear how well ChatGPT performs in the commit message generation task via ICL. Therefore, in this paper, we conduct a preliminary evaluation of ChatGPT with ICL on commit message generation. Specifically, we first explore the impact of two key settings on the performance of ICL on commit message generation. Then, based on the best settings, we compare ChatGPT with several state-of-the-art approaches. The results show that a carefully-designed demonstration can lead to substantial improvements for ChatGPT on commit message generation. Furthermore, ChatGPT outperforms all the retrieval-based and learning-based approaches in terms of BLEU, METEOR, ROUGE-L, and Cider, and is comparable to hybrid approaches. Based on our findings, we outline several open challenges and opportunities for ChatGPT-based commit message generation.