Multimodal Humor Detection Based on Cross-Modal Attention and Modal Maximum Correlation

Multimodal humor detection is an active research area that uses different combinations of modalities for humor understanding user-generated videos. Previous studies often use a single neural network to process multiple modalities together, which ignores the differences between different modalities. Moreover, because there is no effective and sufficient information interaction between neural networks, and thus the complementary information between different modalities is not captured. In this paper, we propose a new humor detection framework, CAMC, which first uses appropriate neural networks for different modalities and then models multimodal interactions at aligned time steps using cross-modal attention networks. A deep canonical correlation analysis is also used in the fusion phase to capture the maximum correlation between the different modalities. Experiments conducted on the publicly available and popular humor detection dataset UR-FUNNY show that our model has significant advantages over previous robust baselines.