Smart Graphical User Interface Component Recognition

Graphical User Interface (GUI) component recognition is one of the most widely used services in mobile test automation. Several industrial and academic solutions are available for GUI component identification. However, the state-of-the-art works are still limited respecting the variety of widgets that can be recognized. Furthermore, most solutions are restricted to widget classification. In this work, we propose SWR, a fully Computer Vision-based method for Smart Widget Recognition. It can recognize 106 GUI component categories including 97 icon classes. Our main contribution is that SWR can generate a detailed description for each identified widget regardless of the component class.