What's Next in my Backlog? Time Series Analysis of User Reviews

User reviews contain valuable information for improving a product e.g., bug reports and feature requests. While large quantities of user reviews for mobile apps are available in app stores, it is very challenging to manually analyze them due to their high volume and existing noise. Therefore, there is a strong need for methods that identify the needle in the haystack, i.e., identify reviews that are worth looking at and that inform the product backlogs with issues to fix or new requirements to consider. Responding to this challenge, we present an approach that identifies such reviews by automatically detecting anomalies (unusual peaks) in time series of user reviews. The approach takes the form of an automatic processing pipeline that ingests user reviews, aggregates them, and produces reports of which aggregates may contain valuable information for software evolution. Both the granularity and sensitivity of the approach can be tuned. With a best-case accuracy and F1-score of 0.88, we show that time-based user review aggregates and automatic anomaly detection serve as a good source of evidence for making estimations as to whether there are events in app store user reviews that warrant the attention of app developers.