Research on Recognition Method of Interface Elements Based on Machine Learning

With the development of mobile Internet, the number of mobile applications has increased dramatically. The core of the mobile application is the interface interaction, and robots can be used to operate mobile applications for GUI automation testing. The recognition of interface elements is the basis of GUI automation test for mobile applications. The GUI models built based on the recognition results of existing interface recognition methods are too simple and the positions of the interface elements identified are not accurate, which cannot meet the testing requirements of mobile application robots. Hence, we propose an improved interface recognition algorithm based on the YOLOv3 algorithm to solve these problems. The experiment results demonstrate that the algorithm can effectively realize the layout classification and region recognition of the interface based on the recognition of interface elements.