A Convolutional Sparse Representations Integration Strategy Based on Self-Attention Genetic Programming for Multimodal Image Fusion

Multimodal image fusion (MIF) seeks to amalgamate complementary information from various image modalities, offering a more comprehensive and accurate representation of data. Convolutional sparse coding (CSC) based methods have demonstrated effectiveness in this domain. However, they encounter difficulties in adaptively discerning and leveraging cross-modality feature correlations, which essentially presents a multi-objective optimization challenge, such as maximizing information retention while minimizing feature distortion. To address this issue, we introduce a self-attention based multi-objective genetic programming (SA-MOGP) method. SA-MOGP frames the quest for an optimal fusion strategy as a multi-objective optimization problem. By integrating the self-attention mechanism into strongly typed genetic programming (STGP), it enables efficient exploration of associations between sparse features across different modalities. Our approach is structured in three hierarchical layers. The feature extraction (FE) layer automatically generates Q, K, V sub-trees according to a predefined function set. These are then input into the self-attention (SA) layer for computation. Finally, the output layer estimates the fusion weights. We utilize the NSGAII framwork to solve this multi-objective problem, aiming for Pareto front solutions. Experiments on infrared-visible and medical image datasets attest to the superior performance of the SAGP method in multimodal image fusion.