Towards Leveraging Semantic Web Technologies for Automated UI Element Annotation

This paper presents a novel Chrome extension for automated web UI element annotation using Semantic Web technologies. The primary objective is to enable Visual Language Models (VLMs) and Large Language Models (LLMs) to quickly comprehend and interact with the web by transforming unstructured visual data into structured semantic information. Our approach integrates natural language processing, vector embeddings, and FAISS for real-time similarity search. In addition to detailing the system architecture and implementation, we propose a custom algorithm for semantic annotation and provide a comparison with traditional methods. Evaluations via BLEU scores and search time measurements demonstrate the extension's efficiency.