Pre-Trained Networks and Feature Fusion for Enhanced Multimodal Sentiment Analysis

Sentiment analysis, a fundamental domain of Natural Language Processing (NLP), is important in diverse applications such as customer service management, social media monitoring, and product review analysis. Nevertheless, as the use of visual data such as photos and videos in online communication continues to grow, conventional text-based sentiment analysis becomes inadequate. Introducing multimodal sentiment analysis is a sophisticated method that uses textual and visual clues to get a more comprehensive knowledge of sentiment. This study introduces an innovative approach to sentiment analysis that combines pre-trained models with a feature fusion of emoticon-based text and picture data. Our algorithm differentiates itself from typical sentiment analysis by using emoticon data and picture features to detect more nuanced sentiment variations, such as extreme positive/negative and neutral. The pre-trained ResNet50 model is used to analyse images from categorised datasets, which include emotions such as Angry, Disgust, Fear, Happy, Neutral, Sad, and Surprise. On the other hand, textual data is processed using the RNN technique in combination with the pre-trained T5 model. In order to enhance efficiency, we use feature fusion algorithms on both modalities, merging the retrieved features from text and pictures to provide a more comprehensive representation. These models, trained on massive datasets for specific tasks, serve as a robust starting point for a model, allowing us to refine and adapt them for multimodal sentiment analysis.