Enhancing VQGAN Performance Through Integration of Multiple Vision Transformers

This paper addresses the problem of improving the initialisation and performance of Vector Quantised Generative Adversarial Networks (VQGANs). Our contribution lies in integrating features from multiple Vision Transformer (ViT) models to create a robust feature extraction framework. These features are clustered using k-means to form an enriched codebook for VQGAN, which enhances the fidelity and stability of generated images. The results of our experiments, conducted on datasets such as Oxford Flowers 102, Oxford-IIIT-Pet, and specific ImageNet-1K categories (birds and architectural structures), demonstrate significant improvements in image quality, as evidenced by lower Fréchet Inception Distance (FID) scores. This methodology advances generative modelling techniques and highlights the potential for enhanced image synthesis through the fusion of ViT and VQGAN models.