A Multimodal Deep Learning Framework for Emotion Recognition in Text and Visual Media

Conventional sentiment analysis techniques mostly depend on textual data, therefore excluding important nonverbal signals such face expressions, environmental images, and video. The multimodal emotion categorization system using textual, visual, and video-based sentiment analysis to overcome this restriction. Three fundamental components make up the system: visual emotion identification using EfficientNetV2-S, textual sentiment analysis utilizing a finetuned GPT-2 model with Low-Rank Adaptation (LoRA), and video-based sentiment tracking employing an EfficientNetB0LSTM hybrid model. Social media postings and captions provide text data; photos and video frames undergo deep learning-based feature extraction. To maximize sentiment categorization, the retrieved embeddings are merged via an attention-based approach. Fine-tuned using LoRA, GPT-2 detects contextual subtleties and sentiment swings, hence enhancing text-based sentiment interpretation. Whereas the EfficientNet B0-LSTM hybrid model examines sequential dependencies in movies to capture dynamic emotional changes, EfficientNet V2-S processes pictures to detect emotions based on face expressions and environmental cues. Dynamic weighting of each modality’s contribution in the fusion module guarantees complete sentiment representation. Real-time chatbot interface deployment of the system allows interactive sentiment analysis for uses including automated customer assistance, mental health evaluation, and social media monitoring. This method surpasses conventional unimodal sentiment analysis methods by combining several modalities, therefore obtaining a deeper and more accurate knowledge of human emotions. A flexible solution for sentiment analysis in many real-world contexts, the suggested system is modular, scalable, and adaptive to future developments in deep learning. Keywords- Multimodal sentiment analysis, deep learning, GPT2 LoRA, EfficientNet, emotion classification, AI-driven chatbot, social media analytics.