MT-Agent: Constructing a GUI Agent via Modality Enhancement and Text-Guided Fusion

Graphical User Interfaces (GUIs) play a crucial role in facilitating user-computer interactions, making them an essential focus of research. However, current automated GUI agents face significant challenges in effectively associating task implementations with specific visual elements, and the resolution constraints of Vision-Language Models (VLMs) also limit the richness of visual information. To this end, we propose a novel multi-modal agent named MT-Agent, which enhances both textual and visual input modalities to enable the model to perceive visual elements in GUIs more effectively. Specifically, Textual Modality Enhancement improves the semantic richness of input text by capturing task-specific details via an external VLM, while Visual Modality Enhancement incorporates fine-grained visual details to better represent critical GUI elements. In addition, we introduce an innovative text-guided directional feature fusion mechanism, which leverages enriched text features to guide the integration with visual information. In experiments, MT-Agent demonstrated exceptional performance on AITZ dataset, achieving an action type prediction accuracy of 84.80% and a step prediction accuracy of 58.07%, surpassing previous state-of-the-art models. Furthermore, on the GUI Odyssey benchmark, MT-Agent achieves performance comparable to previous state-of-the-art models while using only about 1/20 of their trainable parameters. Our codes, demos, and relevant data will be released to facilitate further research and validation within the scientific community.