Branding - Fusion of Meta Data and Musculoskeletal Radiographs for Multi-Modal Diagnostic Recognition

Data fusion techniques provide opportunities for combining information from multiple domains, such as meta and medical report data with radiology images. This helps to obtain knowledge of enriched quality. The objective of this paper is to fuse automatically generated image keywords with radiographs, enabling multi-modal image representations for body part and abnormality recognition. As manual annotation is often impractical, time-consuming and prone to errors, automatic visual recognition and annotation of radiographs is a fundamental step towards computer-aided interpretation. As the number of digital medical images taken daily rapidly increases, there is a need to create systems capable of appropriately detecting and classifying anatomy and abnormality in radiology images. The Long Short-Term Memory (LSTM) based Recurrent Neural Network (RNN) Show-and-Tell model is adopted for keyword generation. The presented work fuses multi-modal information by incorporating automatically generated keywords into radiographs via augmentation. This leads to enriched sufficient features, with which deep learning systems are trained. To demonstrate the proposed approach, evaluation is computed on the Musculoskeletal Radiographs (MURA) using two classification schemes. Prediction accuracy was higher for all classification schemes using the proposed approach with 95.93 % for anatomic regions and 81.5 % for abnormality classification, respectively.