Attention and Meta-Heuristic Based General Self-Efficacy Prediction Model From Multimodal Social Media Dataset

General Self-Efficacy (GSE) is a vital attribute of human psychology that describes one’s belief about his own ability to succeed in specific situations. GSE is composed of cognitive, social, and behavioral skills of an individual. In this research, we first develop a GSE classification model by using Facebook content (i.e., profile photos and statuses). We collect data from a total of 435 Facebook users in an ethical data collection manner. Two hybrid machine learning methods are applied based on distinct feature extraction approaches: tool-based and deep learning-based. In our tool-based approach, we employ Linguistic Inquiry and Word Count (LIWC) and Bidirectional Encoder Representations from Transformers (BERT) for text and Mediapipe and DeepFace for image feature extraction. We apply Particle Swarm Optimization (PSO) for feature selection, resulting in a robust tabular dataset with high predictive performance for GSE scores. In the deep learning-based approach, we apply BERT and 1-dimensional convolutional neural network (1D-CNN) for text feature extraction, while UNet++ handles image segmentation, and VGG16 and ResNet-152 contribute image features, fused via Canonical Correlation Analysis (CCA). We also integrate a co-attention model for image and text features. Traditional machine learning models, including Random Forest (RF), Xgboost, AdaBoost, and Stacking, are then trained on the feature set to predict GSE scores. This comprehensive model showcases a multifaceted approach to GSE prediction, combining tool-based and deep learning methodologies for enhanced accuracy and insights. Then, we develop a GSE prediction model by using the mentioned tool-based (i.e., LIWC, BERT, Mediapipe, and DeepFace) and deep learning-based feature extraction methods from both image and text datasets. The tool-based model achieves remarkable accuracy percentages of 85.80% (text), 91.06% (image), and an outstanding 93.25% for the hybrid model. The deep learning-based model exhibits competitive results, with accuracies of 64.80% (text), 73.06% (image), and 81.87% for the hybrid model.