TCNTransNet: A Semi-Supervised Temporal-Spatial Fusion Framework for Heart Rate Estimation From Camera Video

Remote photoplethysmography (rPPG) offers a remote and non-contact approach to measure physiological signals, such as Heart Rate (HR), from facial videos. With the convenience of capturing video from cameras, rPPG technology can be widely used in daily health monitoring. However, rPPG measurements are susceptible to interference caused by various external factors, such as changes in environmental light and movements from devices and humans. In this paper, we propose a novel rPPG-based HR estimation framework called TCNTransNet. The TCNTransNet is designed with a parallel architecture, which includes Temporal Convolution Network (TCN) and Transformer, can simultaneously capture the local details and global periodicity of rPPG signals. Additionally, we introduce a bidirectional Cross-Attention module to fuse information and mutually complement TCN and Transformer. To leverage the temporal and frequency characteristics of rPPG signals, we adopt a two-stage training strategy in a semi-supervised manner that constrains the signal frequency range learned by the proposed model. We evaluate our method for HR estimation on two challenging public benchmark datasets, COHFACE and VIPL-HR. Extensive experiments demonstrate that our method achieves comparable performance over various complex scenarios.