Midas++: Generating Training Data of mmWave Radars From Videos for Privacy-Preserving Human Sensing With Mobility

Millimeter wave radar is gaining traction recently for enabling privacy-preserving human sensing. However, the lack of large-scale, dynamic radar datasets impedes progress in developing robust and generalized deep learning models for mobile sensing applications. To address this problem, we resort to designing a software pipeline that leverages wealthy dynamic videos to generate synthetic radar data, but it faces two key challenges including i) incorrect camera and human positions leading to erroneous superposition of signal intensity and ii) the signal reflection of the background and humans in mobile scenes. To this end, we design
Midas++
to utilize rich videos to generate realistic radar data via two components: (i) a
human mesh fitting and calibration
component calculates the camera ego-motion parameters to calibrate the extracted human positions; (ii) a
reflection and noise signal estimation
component combines several key modules,
depth prediction
,
reflection model
, and
spatiotemporal noise estimation
, to output coarse radar data, followed by a
U-Net
model to generate realistic radar data. We implement and evaluate
Midas++
with video data from public data sources and real-world radar data, demonstrating that
Midas++
outperforms other state-of-the-art approaches for both activity recognition and object detection tasks.