Evaluating Generative AI Models for Image-Text Modification

Diffusion-based Image Editing models that utilize text prompts and reference images were developed to mitigate the limitations of the text-based image generation models in retaining the generated image’s structure. The domain oversaw the development of several successful image editing models and benchmark datasets that evaluate these models using auto-rating metrics. One critical limitation is that the metrics faced hurdles aligning with human assessments due to the models’ multi-modal nature and unique unexplored challenges. Human alignment is essential to identify bias, measure user satisfaction, and understand context. The prominence of human alignment grows if these models are used in real-world applications and scenarios where ground truths are absent. Through this research, we are introducing a diverse dataset, with the most common image domains and text prompts with varied challenges, and developing a human-evaluation-based framework that accurately analyzes the performance of these models. Additionally, these human-evaluation ratings validate the relevance of the auto-rating metrics, providing insights for future research. Thus, the research qualitatively evaluates several cutting-edge generative AI-based image editing models and auto rating metrics using a newly introduced diversely challenged dataset and human-evaluation framework. The detailed experimental materials is available at https://github.com/princepatrick/image-edit-evaluation