Formal Verification of Synthetic Image Datasets using Large Language Models

The growth in popularity of synthetic image generation through the usage of Artificial Intelligence has garnered interest regarding its efficacy in generating training data for downstream Machine Learning models. However, such synthetic training data must still conform to specified properties to ensure that the downstream models behave as intended, a task that can require manual verification of the correctness of generated datasets. This becomes increasingly crucial in safety critical applications when error tolerance is low. In this paper, we propose a formal approach for verifying the conformity of a synthetic dataset to specifications formalized in Linear Temporal Logic (LTL). The tool, named the Synthetic Data Formal Verifier (SDFV), analyzes an input image dataset and outputs specific LTL violations by leveraging a Large Language Model. Benefits of the tool include time saved when vetting an image dataset and a provided formal check on the dataset that can then be further reviewed. The preliminary experiments with the SDFV result in error rates of $9.5 \%, 14.3 \%$, and $33.3 \%$. However, after modification, we achieved error rates of only $4.8 \%, 4.8 \%$, and $9.5 \%$. Experiments with the SDFV show promise for further work and demonstrate the tool’s ability to both understand and reason about LTLs and synthetic images.