Abstract:
In this article we propose the use of convolutional self-attention for attention-based representation learning, while replacing traditional vectorization methods with a t...Show MoreMetadata
Abstract:
In this article we propose the use of convolutional self-attention for attention-based representation learning, while replacing traditional vectorization methods with a transformer as the backbone of our speech model for transfer learning within our automatic deceit detection framework. This design performs a multimodal data analysis and applies fusion to merge visual, vocal, and speech(textual) channels; reporting deceit predictions. Our experimental results show that the proposed architecture improves the state-of-the-art on the popular Real-Life Trial (RLT) dataset in terms of correct classification rate. To further assess the generalizability of our design, we experiment on the low-stakes Box of Lies (BoL) dataset and achieve state-of-the-art performance as well as providing cross-corpus comparisons. Following our analysis, we report that (1) convolutional self-attention learns meaningful representations while performing joint attention computation for deception, (2) apparent deceptive intent is a continuous function of time and subjects can display varying levels of apparent deceptive intent throughout recordings, and (3), in support of criminal psychology findings, studying abnormal behavior out of context can be an unreliable way to predict deceptive intent.
Published in: IEEE Transactions on Affective Computing ( Volume: 15, Issue: 1, Jan.-March 2024)
Keywords assist with retrieval of results and provide a means to discovering other relevant content. Learn more.
- IEEE Keywords
- Index Terms
- Automatic Detection ,
- Deceit Detection ,
- Transformer ,
- Transfer Learning ,
- Representation Learning ,
- Joint Attention ,
- Correct Classification Rate ,
- Spectroscopic ,
- Ethnic Background ,
- Visual Cues ,
- Emotional Expressions ,
- Language Model ,
- Spatial Attention ,
- Visual Model ,
- Recurrent Unit ,
- Attention Weights ,
- Robust Representation ,
- Cumulative Plots ,
- Action Units ,
- Speech Data ,
- Hidden State Vector ,
- Convolutional Recurrent Network ,
- Micro-expression ,
- Face Orientation ,
- Speech Detection ,
- Subject Baseline ,
- Fusion Mode ,
- Strong Emotions ,
- Neural Network ,
- Facial Expressions
- Author Keywords
Keywords assist with retrieval of results and provide a means to discovering other relevant content. Learn more.
- IEEE Keywords
- Index Terms
- Automatic Detection ,
- Deceit Detection ,
- Transformer ,
- Transfer Learning ,
- Representation Learning ,
- Joint Attention ,
- Correct Classification Rate ,
- Spectroscopic ,
- Ethnic Background ,
- Visual Cues ,
- Emotional Expressions ,
- Language Model ,
- Spatial Attention ,
- Visual Model ,
- Recurrent Unit ,
- Attention Weights ,
- Robust Representation ,
- Cumulative Plots ,
- Action Units ,
- Speech Data ,
- Hidden State Vector ,
- Convolutional Recurrent Network ,
- Micro-expression ,
- Face Orientation ,
- Speech Detection ,
- Subject Baseline ,
- Fusion Mode ,
- Strong Emotions ,
- Neural Network ,
- Facial Expressions
- Author Keywords