VGTest: Automated Testing of Visual Grounding Models

Visual grounding (VG) is a task that involves locating target objects in images based on natural language descriptions. However, due to the complexity of the task, even state-of-the-art VG models may encounter issues with inaccurate localization, which can pose potential safety risks. Therefore, improving the performance and accuracy of VG models is crucial. To address this, we propose an automated testing method called VGTest, designed to evaluate the performance of VG models. VGTest comprehensively considers both visual and linguistic modalities, constructing a disturbance relationship database and designing a smart disturbance selection algorithm to generate diverse derived test cases, thereby revealing potential issues in the models. In our evaluation, we applied VGTest to three advanced VG models. The experimental results show that the test cases generated by VGTest significantly reduced the models' accuracy, with a maximum accuracy drop of 52.38% and an average MultiModal Impact score (MMI) of $\mathbf{3 2. 4 1 \%}$, successfully detecting potential issues in both simple and complex scenarios.