ReChan: An Automated Analysis of Android App Release Notes to Report Inconsistencies

“What's new?” This is what users wonder when they see the notification that a mobile app has just been updated on their device. New releases may involve simple bug fixes, or may include new features that users are eager to try. Regardless of the change, users do want to know what are the differences with respect to the release that have been using so far. The Google Play store has a visible section for each Android app that clearly describes the changes that affect the latest release. This description, however, is curated by developers, and may not match the actual changes in the binary code. This paper presents ReChan, a novel technique aiming to automatically detect mismatches between release notes of Android applications and the actual changes in the code. We define a taxonomy of 9 release categories by manually tagging 1,200 real samples, and we present our solution to automatically classify release notes written in English. ReChan then implements specific analyses to detect such changes in the code, and compares the analyses outcome to detect mismatches. ReChan achieves a precision, recall and f-score of 84.9% on the manually crafted ground truth of three open source apps. Experiments on a dataset of 12,706 closed source Android apps show that developers tend to correctly report changes due to bug fixes and new features, but omit changes that affect the list of requested permissions, the UI and other content that the app uses.