A Comprehensive Study of Bug Characteristics on Foundation Language Models

General Large Language Models (LLMs), such as ChatGPT, have achieved prominent advances in various tasks. In parallel, to better adapt to specific domains, researchers propose multiple domain-specific LLMs by fine-tuning open- source foundation language models (FLMs) (e.g., LLaMA and T5), with immense domain-specific corpora. However, like many other development processes, practitioners may also encounter various bugs. The nature of these bugs, such as their underlying causes, their fixing patterns, and their symptoms remain unclear. Understanding the characteristics of bugs during the utilization of FLMs can help understand, locate, and fix FLM-related bugs, thereby enhancing the quality of FLMs and LLMs fine-tuned upon FLMs. Hence, in this work, we collected 469 real bugs from seven FLMs (T5, OPT, LLaMa, GLM, GPT-NeoX, GPT-J, and Pythia) in GitHub and Stack Overflow. We then manually labeled their root causes, fixing patterns, and symptoms, and also explored the relationship between root causes and fixing patterns. Consequently, we derived 7 significant findings and some key findings include: ❶ the majority of bugs are caused by Dependency-related, API-related, and Code-related issues; ❷ the top three most common fixing patterns in FLM bugs are Update Outdated Dependencies, Modify Configuration, and Add/Remove/-Modify Conditional Expression; ❸ most root causes can have multiple fixing patterns, among them, most bugs caused by API Misuse or API Change can be resolved by Modify Parameter and Replace API; ❹ the majority of these bugs lead to the complete crash of programs. Based on these findings, we shed light on the characteristics of bugs practitioners encountered while utilizing FLMs, pave future research directions, and provide practical suggestions to FLM developers, users, and software engineering researchers.