Deep Neural Networks for Multimodal Sentiment Analysis

Sentiment analysis aims to identify the embedded effective value of the author toward a specific topic from the context. Since sentiments could greatly influence human beings’ daily decision-making and other processes, sentiment analysis could have numerous applications. The multimodal sentiment analysis is an emerging research area in sentiment analysis, which analysis multiple modalities to identify sentiment value. The current unimodal sentiment analysis mainly focuses on text and has been widely used in business fields such as stock analysis. However, due to the development of Internet technology, online context tends to be various in formats: some information is expressed through images or voice. Due to that, multimodal sentiment analysis has attracted much attention in the recent few years and has more potential than unimodal sentiment analysis. However, multimodal sentiment analysis also has more works to be done. Multimodal sentiment analysis not only needs to extract textual features but also needs to extract visual and audio features from the context. Then, multimodal sentiment analysis needs to combine them using proper fusion approaches and identify the correct sentiment value. Plenty of multimodal sentiment analysis works have been done, and various relatively mature multimodal sentiment analysis approaches have been proposed. However, there is still a lack of summary of multimodal sentiment analysis at present. This article reviews the current popular benchmarks of multimodal sentiment analysis and defines several commonly used evaluation criteria. Then, this article retrospect recent developments in multimodal sentiment analysis, including proposed modal fusion approaches. The challenges and possible solutions are also discussed.