Generating Images to Enhance Situational Awareness in Nuclear Power Plants Through Caption-Assisted Computer Vision

Artificial intelligence has made tremendous strides in computer vision and natural language processing, enabling new multimodal systems to fuse these capabilities. This study proposes a multimodal deep learning model to augment images based on descriptive textual prompts. Specifically, the model takes as input an image along with constraints expressed in natural language. It then generates a visually similar new image incorporating the specified textual modifications. For instance, given an image of a train, the model could be instructed to alter the background outside the carriage windows to the night sky. To achieve this, we introduce the Text-based Image Refinement Generator (TIRG) algorithm. TIRG first retrieves semantically similar existing images as references. It then synthesizes a new image integrating the provided text constraints utilizing generative adversarial networks. To evaluate the model, we assess key metrics including similarity between input and output images, accuracy adhering to the textual modifications, and generation speed. Preliminary results indicate TIRG capably fulfills the aim of content-aware image enhancement grounded in descriptive language. Output images visually align with inputs while reflecting textual specifications. This work presents a multimodal deep learning approach tying visual and linguistic modalities to enrich image content through natural language. The proposed algorithm demonstrates promising potential for a variety of enhancement applications.