Synergistic Integration of Image-Text Modalities in Telemedicine: VAE-Infused Graph Convolutional Networks for Diagnostic Decision-Making

The fusion of multimodal data in telemedicine diagnosis plays a crucial role in improving diagnostic accuracy and enabling comprehensive analysis. While integrating multimodal pathological images is relatively straightforward, effectively integrating non-image modalities remains challenging, leading to the underutilization of the complementary characteristics present in multimodal data. We introduce a novel fusion method that addresses this challenge by integrating image and text modalities. Our proposed method leverages a Variational Autoencoder (VAE) to extract hidden states from images. These states are subsequently fused with patient-specific text attributes-age, gender, and diagnostic keywords-during the feature fusion stage. The fused features are structured as a graph and then input into a Graph Convolutional Neural Network (GCN) for classification tasks. Our approach establishes connections between the text and image modalities, exploring the multimodal relationship using GCN. Our method exploits similarities among samples with shared diagnostic keywords, effectively utilizing information present in both modalities. The experimental results show that our fusion scheme outperforms advanced image classification models and multimodal models in terms of classification accuracy. Specifically, in the experiments on the dataset, our method Hidden State Graph Convolutional Network (HSGCN) performed the best in performance metrics such as Precision, Recall, F1 score, Accuracy, and AUC, with Precision reaching 78.42%, significantly higher than other methods. In addition, compared with single modal image classification models and multimodal models, HSGCN also demonstrates higher accuracy in Top-k performance, with average performance for each category exceeding that of single modal classification models, and overall accuracy being 2-4% higher. These results highlight the effectiveness of our method in utilizing supplementary information from image and text patterns to improve remote medical diagnosis.