NeuroUI: A Metamorphic Testing Strategy to Make UI Component Detection Models Robust

Recent advancements in deep neural network have enhanced the capabilities of User Interface (UI) element identification models. However, these models often struggle to accurately identify and classify elements in complex backgrounds and overlapping components. This study proposes NeuroUI, an automated testing approach that introduces structurally consistent Metamorphic Relation (MR) design to make the UI detection models robust. The approach introduces six MRs including Text Language, Text Emoji, Emotive Text Style, Component Occlusion, Geometric Background and Text Overlap which are applied to original seed snapshots to create perturbed snapshots. These are used to form a test oracle where at least one of the elements is classified wrong. When evaluated on four models namely SSD, FCOS, YOLOv5, and Faster R-CNN, NeuroUI achieved an increase in Error Finding Rates (EFR) up to 27.9, 29.4, 34.5, and 32.6 percentage points respectively. Retraining these models with the generated test oracle reduced EFR significantly up to 30.2 percentage points on average for our trained models when tested on 30 real-world applications from the Google Play Store.