Enhancing Image Generation by fusing Auto Encoder & Transformative Generation Approach

Generative Pre-trained Transformer (GPT) models have showcased outstanding ability in language understanding and generation, yet they confront inherent limitations in directly generating high-accuracy images due to their text-based nature. To address this, a novel approach by fusing GPT with Vector Quantized Variational Autoencoders (VQ-VAE) is proposed to overcome the constraints of GPT in image synthesis. This approach helps in using the strengths of VQ-VAE in processing visual data by utilizing it as an encoder to derive discrete latent representations from input images. These latent representations, incorporating essential visual details, are subsequently fed into the GPT decoder. This approach facilitates GPT's interpretation and generation of high-coherence images based on the latent space representations obtained from VQ-VAE. This work acknowledges GPT's limitations as a text-based model and seeks to overcome them by utilizing VQ-VAE's ability to process image data and generate discrete latent codes. By integrating these models, this approach helps in producing images that are both contextually coherent and visually realistic, surpassing the boundaries of GPT's inherent text-centric framework. The proposed generative approach yielded FID scores of 23.35, 28.6 and 21.45 for the MNIST digit classification dataset, CIFAR-10 dataset, and MNIST fashion dataset, respectively