Dependency-aware Form Understanding

Form understanding is an important task in many fields such as software testing, AI assistants, and improving accessibility. One key goal of understanding a complex set of forms is to identify the dependencies between form elements. However, it remains a challenge to capture the dependencies accurately due to the diversity of UI design patterns and the variety in development experiences. In this paper, we propose a deep-learning-based approach called DependEX, which integrates convolutional neural networks (CNNs) and transformers to help understand dependencies within forms. DependEX extracts semantic features from UI images using CNN-based models, captures contextual patterns using a multilayer transformer encoder module, and models dependencies between form elements using two embedding layers. We evaluate DependEX with a large-scale dataset from mobile Web applications. Experimental results show that our proposed model achieves over 92% accuracy in identifying dependencies between UI elements, which significantly outperforms other competitive methods, especially for heuristic-based methods. We also conduct case studies on automatic form filling and test case generation from natural language (NL) instructions, which demonstrates the applicability of our approach.