VMA: Domain Variance- and Modality-Aware Model Transfer for Fine-Grained Occupant Activity Recognition

The growth of the Internet of Things (IoT) sensing systems leads to a large number of multimodal datasets over different deployments. Labeling costs for these datasets, especially fine-grained labels, are often tremendous. On the other hand, different data distributions (domain variance) of these datasets prevent models built with labels of one dataset (source domain) from being directly used in another (target domain). This domain variance may be caused by one or more physical factors change in the deployments, such as buildings and/ or people. Existing model transfer studies mainly focus on adapting the model to the domain variance caused by only one physical factor change. When multiple factors change between the source and target domains, the model transfer often yields low accuracy due to significant domain variance. We present VMA, a model transfer framework for multimodal IoT sensing data that handles multi-factor domain variance. VMA first decouples the multi-factor domain variance between two datasets to multiple single-factor domain variance dataset pairs with other available datasets. Then, VMA leverages sensing modalities robust to each single-factor domain variance for accurate prediction by weighing them more in the fusion. We apply VMA to the fine-grained occupant activity recognition application with a multi-modal sensing system of structural vibration and wearable IMU. We collect real-world datasets to evaluate the proposed framework. VMA achieves a model transfer accuracy up to 76.1% on the target domain with multi-factor domain variance, demonstrating a 1.6x and 1.9x error reduction compared to direct prediction baselines with and without modality-aware learning design.