Using Consumer Camera and Custom Firmware to Monitor Heart Rate in Terminally Ill Children during Music Therapy

Photoplethysmography Imaging (PPGI) is considered a prospective non-contact replacement for existing methods of heart rate (HR) and HR variability monitoring. However, “in the wild” studies are rare. Music therapy for children with severe disabilities aims to alleviate pain and increase their well-being. As they are not able to communicate via spoken language, changes in facial expressions and vital signs may be used to quantitatively assess effects of music therapy. Here, cameras can be used to capture both, hints of the emotional expressions and, via PPGI, HR. From a technical perspective, the high variability in HR, the interaction with the therapist, as well as natural lighting form an ideal scenario to stress-test existing algorithms. We used a consumer camera with custom firmware to record raw videos of children during music therapy. We employ the often used pipeline of face detection, skin segmentation, pulse signal extraction and heart rate estimation. To enforce temporal coherence, we apply temporal filtering to the skin masks inside a 5-frame window. We evaluate the estimated HR using a reference ECG. For the full 30 minute therapy sessions we achieved a mean absolute / root-mean-squared error of 3.98/7.11 BPM. Finally, PPGI methods show promising results for music therapy and can provide accurate HR over a wide range.