Animated Clothing Fitting: Semantic-Visual Fusion for Video Customization Virtual Try-On with Diffusion Models

This work aims to address the new challenge of video customization virtual try-on, enabling the overlay of designated attire onto customized digital character animations. In contrast to conventional video virtual try-on tasks, this approach allows users to individualize their digital avatars based on personal preferences, thus enriching the virtual try-on experience with increased flexibility and engagement. To achieve this objective, the paper introduces an Animated Clothing Fitting (ACF) framework that utilizes a diffusion model integrated with a garment appearance extraction module and a motion control module. This framework not only facilitates the generation of customizable avatar animations but also ensures better preservation of garment consistency. The ACF framework incorporates a garment appearance extraction module to capture the visual aspects of the clothing, infusing the garment details into the video diffusion model from both semantic and visual dimensions, which enhances the retention of the garment's appearance. Following this, a motion control module is devised to accept customized pose animations as input, enabling the adjustment of avatar poses while simultaneously preserving the intricacies of the attire. A multitude of experiments conducted on publicly available datasets have confirmed the exceptional performance of the proposed ACF framework.