A Multimodal Deep Learning Approach Using Late Fusion for Pneumonia Detection Using Transfer Learning

Pneumonia is one of the most fatal respiratory diseases, affecting a large number of people globally every year. The traditional method of pneumonia diagnosis was using chest x-ray images for detecting swelling in lower parts of lungs or detecting presence of any fluid inside lungs but sometimes these common symptoms can be confused with other diseases. Therefore, to address these challenges, modern deep learning-based solutions are emerging as promising solutions. Many existing research studies have highlighted the effectiveness of deep learning models in fields of medical imaging, showcasing the potential of CNNs for automated disease detection. This research paper explores a multimodal deep learning approach for pneumonia detection using Transfer Learning. The model, discussed here, is designed to detect pneumonia related symptoms using chest x - ray images and to observe the abnormalities in breathing sounds, indicating risk of pneumonia. This fusion of visual and acoustic features has proved to strengthen the performance of classification algorithms, making it suitable and useful for real-world clinical use cases. DenseNet121 is used as a pre-trained model for feature extraction from chest x-ray images and mel spectrograms as it outperformed other models like VGG16, VGG19, ResNet50V2, EfficientNetB4, ResNet101V2 and DenseNet201.