Evaluating Generative Language Models with Prompt Engineering for Categorizing User Stories to its Sector Domains

In Agile software development, user stories capture requirements through a concise, user-centric approach. Manual categorization of these stories is both labor-intensive and error-prone. This study addresses the gap in existing research by exploring the application of generative language models, specifically GPT-style models, with prompt engineering for user story categorization.Our research introduces experiments utilizing two generative language models, gpt-3.5-turbo and Llama-2-chat-hf, emphasizing innovative prompt engineering. The evaluation of gpt-3.5-turbo and Llama-2-chat-hf demonstrates commendable performance in zero-shot experiments. Notably, gpt-3.5-turbo outperforms in few-shot prompting with a higher F-score of 67.93%. The gpt-3.5-turbo showcased superior compliance with instructions compared to Llama-2-chat-hf. The study underscores the promising capability of generative language models, with prompt engineering, to automate user story categorization effectively.This work contributes to the potential of automating user story categorization in low-resource settings, showcasing how leveraging LLMs can enhance accuracy and streamline the categorization process in software development, ultimately leading to more efficient outcomes.