Optimizing App Review Classification with Large Language Models: A Comparative Study of Prompting Techniques

This paper explores the application of large language models in classifying app reviews into predefined cate-gories. The study leverages various sizes of Mistral models (tiny, small, medium, and large). It examines the effectiveness of dif-ferent prompting techniques, including zero-shot, one-shot, and few-shot prompting. The research utilizes a dataset of 6406 app reviews, previously categorized into bug/problem report, inquiry, and irrelevant, to evaluate the models' classification performance. Results indicate that few-shot prompting techniques significantly enhance the performance of all model sizes, suggesting that sophisticated prompting can almost offset the limitations of smaller models. The study also finds a diminishing return on investment with increasing model size when applying advanced prompting techniques. This highlights the importance of prompt design in optimizing large language models' performance and suggests that even smaller models can achieve competitive results with the right prompting strategy.