Multimodal Classification of Software Issue Reports Using CLIP: A Fine-Tuned Approach for Bug and Feature Detection

Efficiently identifying software bugs is critical to accelerate software development processes. Traditional methods, which frequently depend on manual review or constraint-driven mechanisms, require a significant amount of time, susceptible to mistakes made by humans, and unable to scale effectively with the growing complexity of modern software systems. With texts, code snippets, and images from well-known repositories like VS Code, Flutter, Roslyn, and Kubernetes in total comprised of 4304 samples, this paper presents a new multimodal approach that uses Contrastive Language-Image Pretraining (CLIP) and its ViT-B/32 variant to effectively address this problem. We experimented with several CLIP variants, including ViT-B/32, RN50, RN101, RN50x4, ViT-B/16, ViT-L/14, and RN50x64, and observed that ViT-B/32 performed well under zero-shot settings, with an accuracy of 56.6%, precision of 54.08%, recall of 53.96% and F1-score of 54.02%, showing the usefulness of general-purpose models in task-specific situations. We performed k-fold cross-validation and hyperparameter tuning on the ViT-B/32 variant, which improved the model’s ability to classify software issues with an 18.8% improvement in accuracy, 19.1% in precision, 20.14% in recall, and 19.6% in F1-score over its zero-shot performance. These findings demonstrate a scalable framework for multimodal classification that goes beyond bugs and features. This framework has potential uses in issue prioritization, automated software maintenance, and productivity tools. These findings emphasize the benefits of using multimodal AI models, such as CLIP, to tackle practical software engineering problems, providing new insights and a solid basis for further developments in this field.