Characterizing and Finding System Setting-Related Defects in Android Apps

Android, the most popular mobile system, offers a number of user-configurable system settings (e.g., network, location, and permission) for controlling devices and apps. Even popular, well-tested apps may fail to properly adapt their behaviors to diverse setting changes, thus frustrating their users. However, there exists no effort to systematically investigate such defects. To this end, we conduct the
first
large-scale empirical study to understand and characterize these
system setting-related defects
(in short as “setting defects”), which
reside in apps and are triggered by system setting changes
. We devote substantial manual effort (
over four person-months
) to analyze 1,074 setting defects from 180 popular apps on GitHub. We investigate the impact, root causes, and consequences of these setting defects and their correlations. We find that (1) setting defects have a wide impact on apps’ correctness with diverse root causes, (2) the majority of these defects (
$\approx$
70.7%) cause non-crashing (logic) failures, and (3) some correlations exist between the setting categories, root causes, and consequences. Motivated and informed by these findings, we propose two bug-finding techniques that can synergistically detect setting defects from both the GUI and code levels. Specifically, at the GUI level, we design and introduce
setting-wise metamorphic fuzzing
, the
first
automated dynamic testing technique to detect setting defects (causing crash
and
non-crashing failures, respectively) for Android apps. We implement this technique as an end-to-end, automated GUI testing tool named
SetDroid
. At the code level, we distill two major fault patterns and implement a static analysis tool named
SetChecker
to identify potential setting defects. We evaluate
SetDroid
and
SetChecker
on 26 popular, open-source Android apps, and they find 48 unique, previously-unknown setting defects. To date, 35 have been confirmed and 21 have been fixed by app developers. We also apply
SetDroid
and
SetChecker
on five highly popular industrial apps, namely WeChat, QQMail, TikTok, CapCut, and AlipayHK, all of which each have billions of monthly active users.
SetDroid
successfully detects 17 previously unknown setting defects in these apps’ latest releases, and all defects have been confirmed and fixed by the app vendors. After that, we collaborate with ByteDance and deploy these two bug-finding techniques internally to stress-test TikTok, one of its major app products. Within a two-month testing campaign,
SetDroid
successfully finds 53 setting defects, and
SetChecker
finds 22 ones. So far, 59 have been confirmed and 31 have been fixed. All these defects escaped from prior developer testing. By now,
SetDroid
has been integrated into ByteDance's official app testing infrastructure named
FastBot
for daily testing. These results demonstrate the strong effectiveness and practicality of our proposed techniques.