Towards High-Fidelity 3D Virtual Try-On via Global Collaborative Modeling

Existing 3D virtual try-on pipelines rely on purely convolutional operations that directly warp the input garment to align with the boundary of the preserved region. However, these methods fail to consider the mutual correlation between the human body and in-shop clothing, leading to texture distortion in the try-on images. Furthermore, these methods face challenges in learning spatial features at the pixel-level representation since they depend on a simple encoder to extract prior information for 3D reconstruction. To address these issues, we propose a Highly Realistic 3D Virtual Try-on (H3DVT+) network. Specifically, we first design a collaborative transformer matching block (CTMB) to establish more reasonable global mutual dependencies that can warp the garment into the spatial distribution under the more natural try-on state. Second, we propose a novel collaborative transformer-based U-Net generating block (CTUGB) using activated context representations to guide the synthesis of more realistic 2D try-on results. Based on these context representations, the significant region within the input data can be enhanced. Finally, we improve a dual-stage 3D reconstruction method: 1) In the first stage, we introduce a multi-resolution parallel network to learn more comprehensive prior features from 2D person images; 2) In the second stage, we introduce a cascaded attention U-Net and a multi-scale discriminator to more accurately infer local fine details of the 3D shape prior for creating fine-grained dressed 3D human body. We conducted performance evaluations on the MPV-3D and VITON datasets, and the experimental results demonstrate that our method outperforms existing state-of-the-art 3D try-on methods in terms of accuracy and realism.