GUI Widget Detection and Intent Generation via Image Understanding

Aerospace control software is the most important part of aerospace software. Since its potential defects endanger life and safety, there are strict requirements on product quality. Therefore, efficient and reliable software testing is essential. The traditional testing method has been challenging to meet its development requirements, and software automation testing has gradually become the main tool for testing aerospace control software. For the automation testing of aerospace control software, the core problem is to locate the GUI widgets on the software screenshots and identify their intent, which directly affects the accuracy of the test. Because of this, we use the widget recognition technology based on image matching and use the image understanding and analysis technology to extract the widget image in the screenshots. After obtaining the widget image, we use a convolutional neural network to extract image features and use the encoder module to encode the extracted information features as a tensor. The decoder module generates a word sequence conditional on tensor and previous output based on the encoded information. We also conduct an empirical study to evaluate the accuracy of widget recognition and intention generation. For widget recognition, our average IoU reached 0.81. For widget intent generation, our model BLEU-1 is 0.567, BLEU-2 is 0.356, BLEU-3 is 0.261, BLEU-4 is 0.131. The results show that our method is very effective.