An Image is Worth Multiple Words: Multi-Attribute Inversion for Constrained Text-To-Image Synthesis

We consider the problem of constraining diffusion model outputs with a user-supplied reference image. Our key objective is to extract multiple attributes (e.g., color, object, layout, style) from this single reference image, and then generate new samples and novel compositions with them. We first perform an extensive attribute distribution analysis that leads to the discovery of an extended conditioning space consisting of multiple textual conditions. These textual conditions vary both per-layer of the U-Net, as well as per-timestep of the denoising process. We observe that although the extended conditioning space provides greater control over different attributes of the generated image, often a subset of these attributes are captured in the same set of U-Net layers and/or across same denoising timesteps. For instance, color and style are captured across same U-Net layers, whereas layout and color are captured across same timestep stages. Existing works in multi-attribute constrained image generation proposed extending textual inversion by learning per-layer or per-timestep tokens, but they suffer from attribute entanglement for the reasons described above. To address the aforementioned gap, we introduce our second contribution where we design a new multi-attribute textual inversion algorithm, MATTE, with associated disentanglement-enhancing regularization losses, that operates jointly across both layer and timestep dimensions and explicitly leads to four disentangled tokens (color, style, layout, and object). We conduct extensive qualitative and quantitative evaluations to demonstrate the effectiveness of the proposed approach.