Multitree Genetic Programming for Multimodal Learning in Multimodal Medical Image Classification

As medical images from multiple modalities provide complementary diagnostic information, multimodal medical image classification (MMIC) leverages their integration to enhance disease diagnosis. Genetic programming (GP), especially multitree GP, offers a flexible framework for evolving multimodal features and holds the potential for interpretability—an essential factor in medical image analysis, where it is crucial to understand the underlying decision-making process. However, current multitree GP methods remain underexplored in MMIC tasks. This article attempts to fill that gap by proposing a novel multitree representation method that enables automatic multimodal feature learning and fusion from different modalities. Based on this new representation, we extend the function and terminal sets to support various types of medical imaging modalities. Additionally, a slice selection factor is proposed to process 3D medical image modalities effectively. Moreover, to fully utilize both modality-specific features and fused multimodal features, fusion is performed at both the feature and decision levels. The accuracy of the proposed method is assessed on eight MMIC tasks, representing different medical scenarios. The new method is compared with 15 competitive algorithms. Experimental results demonstrate that the new method achieves better classification performance than most benchmark methods. Specifically, it achieves the highest accuracy on four out of the eight tasks, GAMMA-Binary, GAMMA-Multi, BraTS2018 and Derm7pt, with improvements of 3.38%, 2.31%, 1.36% and 4.11% respectively over the second-best method. Further analysis reveals that the evolved models show potential for interpretability, owing to their ability to effectively capture the distinctive features of each modality, which may aid in biomarker identification.