Enabling Product Recognition and Tracking Based on Text Detection for Mobile Augmented Reality

We propose a system that supports real-time product recognition and tracking based on text detection for mobile augmented reality. To accurately distinguish products with visually similar packages, we develop a method that recognizes product names by utilizing the characteristics of texts printed on the product packages. It first filters out irrelevant products and effectively ranks candidate products through an inverted index search. We significantly reduce processing overhead by selectively performing product name recognition. In addition, we present an optical-flow-based method that enables efficient and responsive product tracking. Our evaluation shows that the proposed system achieves significantly better product recognition accuracy (80%) compared to alternative solutions, Vuforia (55.4%) and MobileNetV2 (69.6%). We also show that it achieves reasonable tracking accuracy and processing latency to support quality mobile AR experiences.