Unicorn Adapter: Unifying Image and Text Condition Space for Enhanced Diffusion Models

Stable Diffusion demonstrates its proficiency in synthesizing high-quality images based on textual prompts. However, there are cases where textual prompts alone are not enough to convey the exact intent of users. To solve the weak contact between image and text conditions in existing referenced Text-to-Image synthesis, we propose a novel approach of Unicron Adapter (UNIfy features in COmmon RegionN of condition space), which consolidates the image and text conditional spaces by extracting informative image features via cross attention (i.e., Attention Pool) and fusing image and text via self attention (i.e., Fusion Module). Meanwhile, we extend the proposed Unicorn-Adapter for the multimodal semantic concept composition. Our experiments showcase the multimodal conditioning capabilities of our Unicorn-Adapter in image synthesis and highlight the linearity of image embedding within the condition space.