Text-to-Image Diffusion Models via Additive Attention-Based Image Fusion Prompts

In the fine-tuning process of traditional text-to-image models, direct full-parameter fine-tuning of models is not only computationally expensive and impracticable, but also prone to overfitting, weakening the fine-tuned model's generalisation ability, and even triggering catastrophic forgetting in severe cases. In order to mitigate these problems, adapter fine-tuning, as a lightweight solution to achieve efficient fine-tuning by introducing a small number of additional parameters into the pretrained models, has gradually become one of the mainstream methods for fine-tuning text-to-image models. In this paper, firstly, we design an additive attention-based Image Fusion Module (IFM), which enables the adapter to inject two image prompts for training at the same time, enhances the model's ability to extract features from the image prompts. Secondly, the skip connection of the U-Net network in the text-to-image models is used to effectively deliver the image prompts provided by the adapter while reducing the number of adapter parameters. The experimental results demonstrate that our method further improves the performance of the adapter while maintaining the original advantages of the adapter.