LIWC into the Eyes: Using Facial Features to Contextualize Linguistic Analysis in Multimodal Communication

This paper demonstrates that analyzing language patterns in light of their associated facial expressions elicits significant differences between deceptive and truthful communication. Facial Action Units (AU) were analyzed in video recordings (1.2M frames) of 151 dyadic conversations following an interrogation protocol, in which one of the participants is known to be either lying or telling the truth. Linguistic features were extracted from the transcripts using Linguistic Inquiry and Word Count (LIWC) dictionary. Our framework extracted facial-feature contexts automatically corresponding to high and low intensities of AU occurrences. This helped us dive deeper into answers corresponding to the video segments where the witnesses kept their eyes wide open (high intensity of AU05-upper lid raise). We found that in these segments, deceivers used significantly fewer `Seeing', `Perceptual' and `Cognitive' words and their answers were significantly shorter than truth-tellers.