Research on Discernment of Rectangular Controls in Human-Machine Interactive Interface

For the problem of automatic discernment of controls in human-machine interaction interface, the detection and classification methods of controls based on image processing and machine learning are studied. Firstly, image processing algorithm is used to obtain a series of proposal rectangle boxes for human-machine interaction interface. Then, by using the label image in the control, the control category can be identified by machine learning algorithm. In this paper, a data set containing 19 types of control label image is produced, and CNN methods are used to classify them. The experimental results show that the proposal rectangle boxes in the human-machine interactive interface can be captured effectively by combining the image morphology processing algorithm. The rectangular controls can be filtered out by using the label image in the control. The discernment accuracy of CNN model based on VGG16 on verification set can reach more than 97%.