Deep Learning Based Layout Recognition Approach for HMI Software Validation

Human machine interface (HMI) software testing in the automotive industry is a laborious and time consuming process, as testers are required to compare every screen downloaded from the dashboard display cluster against the reference design specified in the requirements. The current industry standard allows for efficient validation of the content of display texts and graphics. However, validating layout information such as the relative positions, alignments, sizes and types of each visual asset on the screen remains a difficult task, as the differences between layouts are not homogeneous. At first glance, some user interfaces may appear to be rather similar but in fact fall under different layouts, due to slight variances in the region or activation conditions. In this work, we propose a two-stage deep learning based framework for efficient layout recognition, which can be applied in general HMI user interface validation tasks. First, we train an object detection model to produce bounding boxes around visual assets within the specified region of interest in the pre-processed input images, along with the object class label. Next, we match the extracted structural information of the test image against the ground truth reference layout designs via two different strategies: a self-supervised learning method (BYOL) and a custom IoU-based layout matching algorithm. While the IoU-based method slightly outperforms BYOL in terms of accuracy, BYOL boasts faster inference speed and can be generalized easily to unseen input images and layouts. Our approach is language agnostic and allows thorough validation of HMI screens in a holistic way.