Digital Twin in Retail: An AI-Driven Multi-Modal Approach for Real-Time Product Recognition and 3D Store Reconstruction

Within large-scale retail environments, adopting a digital twin - an exact virtual model of a physical store - offers substantial opportunities to improve operational efficiency, facilitate decision-making based on data analysis, and enhance the overall customer experience. The current challenges in developing and maintaining a digital twin arise from the labor-intensive processes involved. These include the manual input of product metadata and the tracking of product placements within stores. This issue is particularly pronounced for large retailers, where the complexity escalates due to the sheer volume of products, customers, and field workers. Additionally, discrepancies between store blueprints and their actual layouts, along with seasonal variations and stock shortages, contribute to frequent and significant alterations in the digital twin of the store, making management and updating a challenging task. To solve this, this work proposes an efficient and automated AI-driven multi-modal solution to collect and update on-shelf product data for digital twinning. By taking advantage of cutting-edge head-mounted displays (HMDs) and smartphones, this proposed framework is equipped with robust AI-driven model execution and augmented reality (AR) capabilities, thus facilitating quick data collection and immediate verification. We introduce an AI-driven computer vision system capable of real-time on-shelf product recognition and detecting 2D locations using the mobile device's camera. To gather in-store product location data, we map the 2D product location to the 3D world location, integrating the mobile camera and depth sensor with spatial awareness technology. Augmented reality (AR) is employed on mobile devices to compare the overlay of virtual and physical products, which provides an intuitive solution for verifying the collected product information and in-store location. To validate the effectiveness of the proposed system, we built a product wayfinding iOS app to direct users to the desired product based on our digital twin. We achieved an 81.6% CSAT score and 24.5% NPS score among 46 real shoppers and 23 store associates. Additionally, 91.8% of the testers think the app will make shopping easier and can provide accurate product findings.