A Multimodal Personality Detection Method With Behavioral Action Feature and Association Features *

Multimodal personality detecting aims to identify human personality traits from videos in computer vision. Existing methods mine different modal features separately and fuse them for the Big Five personality detection task, neglecting the mining of multimodal correlation features and behavioral action characteristics. Therefore, we propose a novel multimodal feature learning method for analyzing the personality traits of individuals in short videos. Firstly, we design an Image-Audio Association Feature extraction method (IAAF) to capture the correlation features between images and audio. This method utilizes existing pre-trained models to separately mine image and audio features and calculates the dot product between the two modal features as the association feature. Considering the role of behavioral action features in personality prediction tasks, we utilize pre-trained 3D-ResNet34 as a feature extractor to mine this feature. Meanwhile, We use CLIP to extract the image-personality descriptions feature between images and personality descriptors. This paper uses a pre-trained Roberta to extract text features to enhance the method’s robustness. Finally, the proposed method is tested on two benchmark datasets, and it outperforms existing state-of-the-art methods.