MMW-Carry: Enhancing Carry Object Detection Through Millimeter-Wave Radar–Camera Fusion

This article introduces MMW-Carry, a system designed to predict the probability of individuals carrying various objects using millimeter-wave (MMWave) radar signals, complemented by camera input. The primary goal of MMW-Carry is to provide a rapid and cost-effective preliminary screening solution, specifically tailored for non-super-sensitive scenarios. Overall, MMW-Carry achieves significant advancements in two crucial aspects. First, it addresses localization challenges in complex indoor environments caused by multipath reflections, enhancing the system’s overall robustness. This is accomplished by the integration of camera-based human detection, tracking, and the radar–camera plane transformation for obtaining subjects’ spatial occupancy region, followed by a zooming-in operation on the radar images. Second, the system performance is elevated by leveraging long-term observation of a subject. This is realized through the intelligent fusion of neural network results from multiple different-view radar images of an in-track moving subject and their carried objects, facilitated by a proposed knowledge-transfer module. Our experiment results demonstrate that MMW-Carry detects objects with an average error rate of 25.22% false positives and a 21.71% missing rate (MR) for individuals moving randomly in a large indoor space, carrying the common-in-everyday-life objects, both in open carry or concealed ways. These findings affirm MMW-Carry’s potential to extend its capabilities to detect a broader range of objects for diverse applications.