Identity Anchoring: Preventing Semantic Drift in Personalized Generation with Per-Timestep Attention Normalization

Personalized text-to-image generation faces a critical challenge: during the iterative denoising process of diffusion models, a subject’s core identity features often degrade, a phenomenon we term "semantic drift." This drift results in a noticeable loss of fidelity in the final generated image. To address this, we introduce the concept of Identity Anchoring and propose a novel, plug-and-play Identity Anchor Module (IAM). The IAM calibrates the generation trajectory in real-time by anchoring the process to the reference identity at each denoising step. Specifically, for any given timestep, it extracts the mean and standard deviation from the self-attention maps of the noised reference image. These statistics are then used to normalize the intermediate features of the image being generated, effectively preventing the subject’s characteristics from deviating. As a training-free plugin, IAM seamlessly integrates into existing pre-trained diffusion models. Experiments demonstrate that our IAM module significantly improves subject fidelity, effectively suppresses semantic drift, and achieves excellent performance across multiple quantitative metrics.