Multimodal Automatic Personality Perception Using ViT, BiLSTM and VGGish

Automatic personality perception has become one of the hot research topics in the fields of psychology and affective computing in recent years. The core issue to be addressed in this research topic is how to accurately analyze an individual's psychological characteristics through their first-impression behaviors. To fully utilize the spatiotemporal information of audio and video modalities, this paper proposes a multimodal automatic personality perception method by integrating advanced deep neural network models including ViT, BiLSTM, and VGGish. In terms of video modality, the hierarchical ViT is used to extract effective features from video frames, and then BiLSTM is employed to capture the spatiotemporal information between the video frames. For the audio modality, the pretrained VGGish network is used to automatically obtain advanced audio features. Finally, the features learned from two modalities are integrated to achieve automatic recognition of personality traits. Comparative experimental results on the ChaLearn First Impressions V2 dataset show that the method proposed in this paper outperforms other methods in terms of Pearson correlation coefficient and average accuracy.