“Your Behavior Makes Me Think It Is a Lie”: Recognizing Perceived Deception using Multimodal Data in Dialog Games

Deception occurs frequently in our life. It is wellknown that people are generally not good at detecting deception, however, behaviors of interlocutors during an interrogatordeceiver conversation may indicate whether the interrogator thinks the other person is telling deceptions or not. The ability to automatically recognize such a perceived deception using behavior cues has the potential in advancing technologies for improved deception prevention or enhanced persuasion skills. To investigate the feasibility to recognize the perceived deception from behaviors, we utilize a joint learning framework by considering acoustic-prosodic features, linguistic characteristics, language uses, and conversational temporal dynamics. We further incorporate personality attributes as an additional input to the recognition network. Our proposed model is evaluated on a Daily deceptive dialogue corpus of Mandarin database. We achieve an unweighted average recall of 86.70% and 84.89% on 2-class perceived deception-truth recognition tasks given the deceiver is telling either truths or lies, respectively. Further analyses unveil that 1) the deceiver's behaviors affect the interrogator's perception (e.g., the higher intensity of the deceiver makes the interrogator believe their statements even though they are deceptive in fact), 2) the interrogator's behavior features carry information about their own deception perception (e.g., interrogator's utterance duration is correlated to his/her perception of truth), and 3) personality traits indeed enhance perceived deception-truth recognition. Finally, we also demonstrate additional evidence indicating that human is bad at detecting deceptions - there are very few indicators that overlaps between perceived and produced truth-deceptive behaviors.