Generative AI Models for Automated User Review Feedback on Digital Platforms

The rise of digital platforms makes it increasingly challenging for developers to manage the growing number of user reviews. Replying to each review manually is time-consuming, and template responses often fail to deliver personalized and natural responses. With this challenge, we explore the use of generative AI models that can generate contextually appropriate and personalized responses by learning from data through fine-tuning. This allows for greater variety in responses, avoiding the repetition of static templates. In this study, we evaluated three lightweight and publicly available models: T5-small, FLAN-T5-small, and BART-base, for comparison. These models were chosen for their ability to achieve strong performance while being computationally efficient, which is particularly important when working with moderate-sized datasets. The dataset used consists of 10,000 review-response pairs scraped from Spotify and the Google Play Store and is split into training, validation, and test sets with an 80-10-10 ratio. Our results show that T5-small achieved the highest scores in automatic evaluation metrics, with a BLEU-1 score of 0.5095, a BLEU-4 score of 0.3344, and a ROUGE-L score of 0.4606. However, human evaluations revealed that BART-base outperformed the others in fluency, relevance, and accuracy, scoring 4.37, 3.73, and 4.03, respectively. FLAN-T5-small without instructions also performed well, while its performance with instructions was notably lower, highlighting the importance of prompt design in generative AI models. These findings underscore the challenges of deploying generative AI at scale and emphasize the importance of integrating automated metrics with human insights to generate contextually relevant and natural responses.