Text-Image Data Classification and Cross-Modal Retrieval Method Using Feature Fusion

In this paper, we present a text-image data classification and cross-modal retrieval method using feature fusion. Features from various data types are obtained by extracting their embedding vector through multiple encoders. A contrastive learning process is added while training the network to make different features similar. The extracted features are generated as high-quality fused features through cross-attention of the transformer. The proposed method is evaluated, and it shows that text-image data classification performance was higher than the method using a single data type. We also confirmed that the cross-modal retrieval method using the image and text encoders of the trained network showed high performance.