Segmentation and Multi-Class Classification of GUI/UI Textual and Non-Textual Elements Through Enhanced AI Framework

The process of building and testing Graphical User Interface (GUI) elements in Android apps is labor-intensive, requiring meticulous validation by human testers. This work proposes an automated framework for extracting, preprocessing, and classifying GUI elements from mobile app screenshots. Using OpenCV, the system preprocesses images by removing device-specific artifacts, and a Convolutional Neural Network (CNN) classifies non-textual elements. Optical Character Recognition (OCR) integration enhances text analysis by addressing variations in font size and orientation. The framework generates a JSON file with the semantic properties of each element. Extensive experimentation on a large-scale mobile app dataset demonstrates the framework’s effectiveness, achieving a training accuracy of 97.61%. Compared to traditional GUI analysis tools, the proposed method significantly improves accuracy, scalability, and automation. Key results include enhanced cross-platform compatibility, precise element recognition, and streamlined mobile app testing workflows. This hybrid framework, combining CNN and OCR, provides developers and QA teams with a robust tool to automate UI analysis, offering significant improvements in efficiency, reliability, and user-focused design.