IR-Based Sleep Monitoring: Movement, Respiration, and Sleep Staging With a Structure-Aware Cross-Modal EEG Knowledge Distillation

It is inevitably crucial to classify the sleep stage for assessing sleep quality and diagnosing related diseases. However, existing automated diagnosis methods mostly adopt the gold-standard electroencephalogram (EEG) or other sensing signals from the Polysomnography (PSG) machine in hospital, which are expensive, importable, and therefore unsuitable for point-of-care monitoring at home. To enable the sleep stage monitoring at home, in this article, we integrate the external visual information from infrared (IR) videos with internal bio-electrical knowledge from EEG signals, and propose a novel IR-based sleep monitoring system in Internet of Things (IoT) scenarios: which can noncontact monitor and predict body movement, respiration rate, and sleep staging tasks based solely on IR video. It is different from previous video classification and multimodal analysis systems, mainly in that 1) the temporal duration of the IR video is relatively long, reaching 10 h per night and 2) the semantic gap between the EEG signal and IR video is disparate and much larger than conventional cross-modal data in multimedia analysis, such as video and audio. To establish a solid cross-modal benchmark in sleep monitoring, we develop a new dataset $(\text {S}^{3}\text {IE})$ , which is a large-scale dataset, including synchronized IR videos and EEG signals. We also propose a novel cross-modal knowledge distillation (KD) method, namely, the structure-aware contrastive distillation (SACD) to distill the EEG bio-electrical knowledge to IR visual features. Our SACD achieves the SOTA performances on both our $\text {S}^{3}\text {IE}$ and other cross-modal distillation benchmarks. We expect to raise more attention and promote more developments for the sleep and, more importantly, the cross-modal distillation from clinical signal/media to conventional media in IoT scenarios.