DiffPPO: Reinforcement Learning Fine-Tuning of Diffusion Models for Text-to-Image Generation

In recent years, significant progress has been made in the field of text-to-image generation, particularly with the application of diffusion models. However, existing diffusion models still face challenges in accurately understanding complex textual inputs, precisely controlling object attributes and quantities in the generated images, and maintaining consistency within the scene. To address the limitations of existing diffusion models in generating high-quality images that align with textual prompts, we propose a method called DiffPPO. This approach fine-tunes the diffusion model using the Proximal Policy Optimization (PPO) algorithm and models the denoising process as a Markov Decision Process (MDP). We introduce an Actor-Critic architecture with a value network to reduce the variance of the policy gradient. Additionally, we employ dynamically adjusted KL divergence regularization and entropy regularization to balance the exploration of the image generation policy with the consistency of the pretrained model. Experimental results show that DiffPPO outperforms traditional supervised fine-tuning and reinforcement learning methods in generating images with complex attributes such as shape, emotion, quantity, and artistic style, demonstrating its effectiveness.