MovingFashion: a Benchmark for the Video-to-Shop Challenge

Retrieving clothes that are worn in social media videos (Instagram, TikTok) is the latest frontier of e-fashion, referred to as "video-to-shop" in the computer vision literature. In this paper, we present MovingFashion, the first publicly available dataset to cope with this challenge. MovingFashion is composed of 14855 social videos, each one of them associated with e-commerce "shop" images where the corresponding clothing items are clearly portrayed. In addition, we present a novel baseline for this scenario, dubbed SEAM Match-RCNN. The model is trained by image-tovideo domain adaptation, allowing the use of video sequences where only their association with a shop image is given, eliminating the need for millions of annotated bounding boxes. SEAM Match-RCNN builds an embedding, where an attention-based weighted sum of few frames (10) of a social video is enough to individuate the correct product within the first 5 retrieved items in a 14K+ shop element gallery with an accuracy of 80%. This provides the best performance on MovingFashion, comparing exhaustively against the related state-of-the-art approaches and alternative baselines
¹
.