Feature Stacking Fusion in Multimodal Neural Architecture Search

Multimodal neural architecture search is a crucial task that employs the automatic search approach to find the best multimodal neural network for alleviating the burden of manual design. However, previous works have focused on the selection of cell input features and the search for cell operations, ignoring the fusion effect between modalities. Therefore, to better integrate multimodal information, we propose a multimodal neural architecture search method with feature stacking fusion. Firstly, we use the unimodal model to extract low-dimensional features and high-dimensional features of text and images. Next, two independent cells are used to fuse the low-dimensional features and high-dimensional features of different modalities, respectively, and then a hybrid cell is used to integrate the outputs of the previous two cells for capturing the global multimodal correlation. We conducted experiments on two text-image multimodal classification tasks, and the results show that our model exhibits more competitive performance compared to NAS-based multimodal fusion methods.