Time Awareness in Deep Learning-Based Multimodal Fusion Across Smartphone Platforms

Many modern IoT applications integrate smartphones in their deployments to capture and interface with various sensors. Fusing across sensing modalities can often result in improved application performance. An essential yet often overlooked requirement in multimodal fusion is the synchronization across input data streams. Given the universality of the smartphone system clock, it is commonly used to timestamp sensing data. Through a systematic study of system clock accuracy on modern smartphones, we show that there are drastic timing errors ranging from milliseconds to multiple seconds. Thus, when fusing data derived from multiple smartphones, input data streams may have significant synchronization errors. Although deep learning classifiers for multimodal fusion can achieve state-of-the-art accuracy, the impact of timing errors has yet to be characterized. In this paper, we quantify the impact of timing errors on a multimodal fusion classifier for human activity recognition. Our results indicate that data sync errors of 600ms can degrade classifier accuracy by 3%, and the clock errors such as those observed in Android can result in accuracy drops of up to 25%. To mitigate these concerns, we suggest an application-level system clock replacement and introduce a novel data augmentation technique that improves the evaluated classifier resilience to timing errors.