Enhance Chest X-ray Classification with Multi-image Fusion and Pseudo-3D Reconstruction

Chest radiography (X-ray) is a critical imaging modality for the diagnosis of thorax diseases. Automated classification of chest X-rays has enormous potential to benefit diagnostic decision making. Most existing models are designed to consider a single-view chest X-ray image, mostly the frontal view, failing to utilize the complementary information provided by the lateral view when available. In clinical practice, radiologists often examine both frontal and lateral views to gather more information for diagnosis. Thus, a well-designed model should take advantage of the paired views to enhance the classification results. To this end, we present a novel dual-view deep learning framework to enhance the classification that uses an intermediate bi-directional fusion architecture to exploit the intrinsic spatial correlation registered between the two views. In particular, we design two different modules, a vertical alignment fusion and a 3D reconstruction fusion, that can effectively extract and fuse important features from both input streams to generate a pair of recalibrating signals, which are then used to amplify the pertinent parts while suppressing the irrelevant segments of the feature map in each stream. We evaluate our proposed methods extensively on two large public chest X-ray datasets, CheXpert and MIMIC-CXR-JPG. Experiment results show that our methods can effectively improve the classification results over fine-tuned baselines and other state-of-the-art methods, demonstrating the benefit of exploiting the spatial correlation between the paired views in the model. We also find that our fusion methods significantly outperform the early fusion and late fusion models, highlighting the need to capture the spatial correlation across different intermediate feature layers.