Multi-Modal Language Analysis with Hierarchical Interaction-Level and Selection-Level Attentions

As an emerging research area in natural language processing, multi-modal human language analysis spans language, vision and audio modalities. Understanding multi-modal language requires not only the modeling of independent dynamics within each modality (intra-modal dynamics), but also more importantly interactive dynamics among different modalities (inter-modal dynamics). In this paper, we propose a hierarchical approach to multi-modal language analysis with two levels of attention mechanism, namely interaction-level, which captures the intra-modal and inter-modal dynamics across different modalities with multiple types of attention, and selection-level attention, which selects the effective representations for final prediction by calculating the importance of each vector obtained from interaction-level. Empirical evaluation demonstrates the effectiveness of our proposed approach to multi-modal sentiment classification, sentiment regression and emotion recognition.