VTONShoes: Virtual Try-on of Shoes in Augmented Reality on a Mobile Device

The virtual try-on (VTON) system in augmented reality (AR) has attracted significant research interest. This paper presents a novel real-time AR virtual shoe try-on system (VTONShoes). Users can see the virtual shoes with full degrees of freedom on the mobile device. In particular, we propose an efficient framework to detect, classify, and recover 6-DoF pose of shoes from the captured images and then accurately render the 3D shoe model on the screen in realtime and in full degrees of freedom. For accurate pose recovery, dense keypoints are designed on the 3D shoe model. An efficient joint 2D keypoint localization and leg silhouette segmentation module (KeyPointLoc) is designed to predict keypoint projections on 2D images and the shoe-leg occlusion relationship. In order to reduce jitter between frames, an optimization-based framework with the longest continuous invariant subarray constraints is proposed to minimize classification errors caused by model switching, and a smoothing module with Exponential Weights Decay is presented to post-process the rendered results. We also developed a large-scale dataset named Diverse-Shoes which contains images extracted from 80K videos, annotated with the shoe bounding box, transformation matrices, and silhouettes of legs. Our system has achieved a smooth and stable try-on effect on mainstream devices with a real-time speed of around 25 to 45 FPS on mainstream mobile phones, which significantly outperforms state-of-the-art methods for real-time performance and rendering accuracy.