Fully Convolutional Hybrid Fusion Network With Heterogeneous Representations for Identification of S1 and S2 From Phonocardiogram

Heart auscultation is a simple and inexpensive first-line diagnostic test for the early screening of heart abnormalities. A phonocardiogram (PCG) is a digital recording of an analog heart sound acquired using an electronic stethoscope. A computerized algorithm for PCG analysis can aid in detecting abnormal signal patterns and support the clinical use of auscultation. It is important to detect fundamental components, such as the first and second heart sounds (S1 and S2), to accurately diagnose heart abnormalities. In this study, we developed a fully convolutional hybrid fusion network to identify S1 and S2 locations in PCG. It enables timewise, high-level feature fusion from dimensionally heterogeneous features: 1D envelope and 2D spectral features. For the fusion of heterogeneous features, we proposed a novel convolutional multimodal factorized bilinear pooling approach that enables high-level fusion without temporal distortion. We experimentally demonstrated the benefits of the comprehensive interpretation of heterogeneous features, with the proposed method outperforming other state-of-the-art PCG segmentation methods. To the best of our knowledge, this is the first study to interpret heterogeneous features through a high level of feature fusion in PCG analysis.