SatiIndicator: Leveraging User Reviews to Evaluate User Satisfaction of SourceForge Projects

Quality of software (QoS) is important for users, as it may lead to high cost when a user or a company happens to pick up a software project with low quality. In recent years, many software quality assessment models take user satisfaction as an important metric for measuring software quality. However, user satisfaction on a software project is usually not precisely evaluated. In this paper, we propose a novel, automated approach called SatiIndicator to evaluate user satisfaction of a software project by analyzing user reviews with user opinions and emotions. The essential idea of SatiIndicator is to (1) use a topic model to cluster all aspects in a software genre into different topics and compute the weight of each topic, (2) take sentiment analysis and calculate the sentiment strength of every aspect and pure attitude reviews, and (3) evaluate the user satisfaction score for a software project. Wilson Interval is applied to punish the software projects with insufficient reviews in order to keep fairness. We have evaluated SatiIndicator on ten software genres in SourceForge. The evaluation results show that when software projects have sufficient reviews, SatiIndicator performs 35% higher than baselines at p@3, 15% higher than baselines at p@15 and over 85% Spearman Coefficient with the ground truth. When software projects have insufficient reviews, SatiIndicator performs 30% higher than baselines at p@3, 15% higher than baselines at p@15 and over 60% Spearman Coefficient with the ground truth.