User Interface Code Retrieval: A Novel Visual-Representation-Aware Approach

Code reuse has been perceived to be an effective tool for user interface development that is known to be complex and messy. A fundamental problem in user interface code reuse is how to effectively retrieve working code from existing code repositories, which renders functional interfaces similar to a programmer's input sketch. Existing generic code retrieval techniques are normally inadequate for user interface code due to its associated visual information. In this paper, we put forward a novel approach that retrieves matching candidate code of an interface sketch based on their visual-representation trees. To better capture sub-region similarities, we propose to decompose a visual-representation tree into an ordered list of paths (or sequences). We then devise a novel algorithm to measure the similarity between two sequences by leveraging various visual information and another algorithm to calculate the similarity of two ordered lists by considering the relative positions of different widgets. To evaluate our solution, we carefully design an experiment to generate the similarity scores of 6,750 user interface pairs based on real users' judgement. Based on this ground truth, we show that our method is able to generate accurate similarity scores, outperforming several state-of-the-art competitors.