Human-MoE: Multimodal Full-Body Human Image Synthesis with Component-driven Mixture of Experts

Conditional full-body human synthesis is to generate and edit realistic images based on given conditions. Previous methods lay a solid foundation but may have limitations in adjusting human poses and appearance. They usually adopt a monolithic design, and the details of generated images are prone to be indistinct or distorted due to the high variation of human appearances. To tackle the above-mentioned challenges, we propose Human-MoE for multi-modal full-body human synthesis. Users can control the image generation through three types of input representations: parsing maps for geometry, text descriptions for appearance attributes, and pose maps to distinguish postures. Our framework specifically designs a mixture-of-experts module to capture and synthesize details in specific regions with high fidelity. These synthesized details are then applied to refine the appearance. Our method achieves top scores in experiments by multiple metrics, especially FID and SSIM, demonstrating its advance in visual quality and controllability.