An Improved Icon Matching Method for GUI Testing of Mobile Devices

In recent years, with the emergence of Industry 4.0, automated testing technology based on machine vision has been widely used in intelligent manufacturing production lines. However, in the visualization test task of mobile devices, due to the interference of the complex environment of the factory and the problem of the feature images of the GUI controls not being obvious, As a result, when industrial robots perform matching and positioning operations on GUI controls on mobile devices, matching failures and mismatching occur. Firstly, this paper proposes a feature extraction method that combines edge detection algorithm, sift algorithm, and DBSCAN clustering algorithm to extract the feature information of all GUI accurately controls on the mobile device's screen to be detected for further image matching. Secondly, an improved twin neural network is proposed to solve the problem of recognizing template icons and icons to be tested. Based on the original model, the GeLU activation function is introduced by replacing the backbone structure of the Rep VGG network and adding an attention mechanism to the feature extraction part. The method was evaluated on a specially constructed GUI control icon dataset, and the accuracy of the improved network model reached 98.6%. When performing the GUI control matching task, the total elapsed time for matching is much shorter than other existing algorithms, and the correct rate of matching to the correct icon in the batch test also reaches 94.4%. Applying the proposed matching method to the visualization testing task effectively improves the matching accuracy of the robotic arm to the GUI control icons on the mobile device screen.