Rethinking Multimodality: Optimizing Multimodal Deep Learning for Biomedical Signal Classification

This study presents a novel perspective on multimodal deep learning for biomedical signal classification, systematically analyzing the impact of complementary feature domains on model performance. While fusing multiple modalities often presumes enhanced accuracy, this research demonstrates that adding modalities can yield diminishing returns, as not all fusions are inherently advantageous. To validate this, five deep learning models were rigorously evaluated for electrocardiogram (ECG) classification: three unimodal models (1D-CNN for time, 2D-CNN for time-frequency, and a 1D-CNN-Transformer for frequency) and two multimodal models. Our findings, supported by Bayesian inference and p-values <0.05, show that the fusion of the time and time-frequency domains consistently outperformed the unimodal baselines. This confirmed the synergistic complementarity of these specific domains. Conversely, including the frequency domain offered no further improvement and, at times, led to a marginal decline in performance, a phenomenon we substantiated with an ablation study. This indicates that the frequency domain provided representational redundancy rather than new, useful information. Redefining the fundamental principles of multimodal design in biomedical signal analysis, we introduce the Complementary Domain Theory, which posits that optimal domain fusion is not about the number of modalities but the quality of their inherent complementarity. This paradigm shifts the focus from traditional, heuristic feature selection to a mathematically quantifiable framework for identifying ideal domain combinations. Our findings, while applied to ECG, have broader relevance to other biomedical domains involving multimodal time-series data. The core insight is that optimal multimodal performance arises from the intrinsic information-theoretic complementarity among fused domains.