HBFNet: A Two-Stage Heterogeneous Network For Multimodal Emotion Fusion

Modal fusion technology plays a key role in the utility of multimodal sentiment analysis. Because attention models the dense correlation of sequences, it is competitive and robust in related tasks. However, these methods ignore the spatial density of different modes. While simultaneously processing and fusing visual, audio and text, they not only introduce a large number of low-order redundant features, but also sacrifice a large amount of computational cost. Therefore, we propose a Heterogeneous Bottleneck Fusion Network (HBFNet), After the redundant features are reduced, the refined higher-order features are fused by a small “fusion bottleneck”. The HBFNet uses small computational cost to refine the low-order dense fine-grained information in low-density modes, and completes the interactive fusion of different modes in different stages. In order to verify the effectiveness of the proposed method, we do experiments on CMU-MOSEI datasets respectively. We find that this fusion method improves the fusion performance and reduces the number of network parameters, and the proposed method is superior to state of the art methods.