Multimodal Sentimental Analysis Using GloVe Convolutional Neural Network with Capsule Network

Leverage the power of deep learning-based Multimodal Sentiment Analysis to gain a deeper understanding of emotions and sentiments in various forms of data, including text, images, and video. This research is conducted on the CMU-MOSI and CMU-MOSIE datasets, which comprise text, audio, and video clips. In this paper, propose the utilization of GloVe for text feature extraction and sentiment classification (negative, positive, and neutral). For audio feature extraction, employ Mel-Frequency Cepstral Coefficients (MFCC) and utilize a Convolutional Neural Network (CNN) for audio sequence classification. Additionally, incorporate the Capsule Network for video clip classification, effectively addressing limitations seen in previous models. As a result, of the model achieves outstanding performance metrics on the CMU-MOSI dataset, with ACC2 at 86.95% and ACC7 at 56.95%, and on the CMU-MOSIE dataset, it attains ACC2 at 87.55
^%
and ACC7 at 55.80%, surpassing the capabilities of existing models such as Hybrid Contrastive Learning of Tri-Modal Representation (Hycon) and (MMLATCH) Bottom-Up Top-Down Fusion.