GUI Element Detection Based on YOLOv5s

The identification of GUI elements in a user interface design diagram is fundamental to many software engineering tasks. In order to solve the problems of low detection accuracy of GUI element recognition, low recognition accuracy and systematic extraction of GUI element information, this paper proposes a GUI element extraction method based on user interface design diagram. Firstly, the improved object detection model YOLOv5s-Shuffle-CIOU was used to detect GUI elements in the image user interface. For GUI elements classified as text elements, the optical character recognition (OCR) model is used for character extraction. Finally, relative positions of text GUI elements and non-text GUI elements within the graphical user interface are performed, and the resulting integration is encapsulated as an integrated JSON object representing GUI elements. This method can effectively improve the detection and classification accuracy of different categories of GUI elements, and can systematically extract all GUI element attribute information in graphical user interface. This approach is platform-independent and helps with model building in code generation and GUI testing.