Noncontact Heart Rate Estimation via Space–Time Video Super-Resolution for rPPG Information Recovery

Heart rate is a vital physiological indicator that can be estimated in a noncontact manner through remote photoplethysmography (rPPG) using facial videos. Video quality is essential for the extraction of the rPPG signal and the accurate estimation of heart rate. Insufficient frame rate and resolution can cause loss of rPPG information, compromising heart rate estimation accuracy. To address this problem, a two-stage noncontact heart rate estimation method is proposed, which incorporates space-time video super-resolution (STVSR) technology for low-frame-rate (LFR) and low-resolution (LR) video scenes. In the first stage, an STVSR network (FrameReconNet) and an rPPG information recovery network (rPPGRecovNet) are cascaded to achieve simultaneous video reconstruction and rPPG information recovery, guided by a hybrid loss function. In the second stage, an end-to-end network (rPPGExtractNet) is used to extract rPPG signals from videos reconstructed by FrameReconNet and estimate the final heart rate values. To address the limitation of the original STVSR network, which extracts only local features from short frame sequences and leads to rPPG discontinuity between adjacent sequences, a novel local-global-local (LGL) temporal feature aggregation and partition operation is proposed. This ensures consistent global temporal information across all frame sequences in a facial video. The experimental results on the VIPL and multimodal multivital sign (MMVS) datasets demonstrate that the proposed method is an effective noncontact heart rate estimation method, which can recover rPPG information from LFR and LR facial videos. The proposed method achieves comparable heart rate estimation accuracy with the fewest parameters, balancing the requirements of accuracy and real-time performance.