Abstract:
Studies of time-continuous human behavioral phenomena often rely on ratings from multiple annotators. Since the ground truth of the target construct is often latent, the ...Show MoreMetadata
Abstract:
Studies of time-continuous human behavioral phenomena often rely on ratings from multiple annotators. Since the ground truth of the target construct is often latent, the standard practice is to use ad-hoc metrics (such as averaging annotator ratings). Despite being easy to compute, such metrics may not provide accurate representations of the underlying construct. In this paper, we present a novel method for modeling multiple time series annotations over a continuous variable that computes the ground truth by modeling annotator specific distortions. We condition the ground truth on a set of features extracted from the data and further assume that the annotators provide their ratings as modification of the ground truth, with each annotator having specific distortion tendencies. We train the model using an Expectation-Maximization based algorithm and evaluate it on a study involving natural interaction between a child and a psychologist, to predict confidence ratings of the children’s smiles. We compare and analyze the model against two baselines where: (i) the ground truth in considered to be framewise mean of ratings from various annotators and, (ii) each annotator is assumed to bear a distinct time delay in annotation and their annotations are aligned before computing the framewise mean.
Published in: IEEE Transactions on Affective Computing ( Volume: 9, Issue: 1, 01 Jan.-March 2018)
Funding Agency:
Department of Electrical Engineering, University of Southern California, Los Angeles, California
Rahul Gupta received the BTech degree in electrical engineering from
Indian Institute of Technology, Kharagpur, in 2010 and the PhD degree in electrical engineering from the University of
Southern California (USC), Los Angeles, in 2016. His research concerns development of machine learning algorithms with
application to human behavioral data. His dissertation work is on the development of computational methods for
mod...Show More
Rahul Gupta received the BTech degree in electrical engineering from
Indian Institute of Technology, Kharagpur, in 2010 and the PhD degree in electrical engineering from the University of
Southern California (USC), Los Angeles, in 2016. His research concerns development of machine learning algorithms with
application to human behavioral data. His dissertation work is on the development of computational methods for
mod...View more
Watson Group, International Business Machines, Yorktown Heights, New York
Kartik Audhkhasi received the BTech degree in electrical engineering
and the MTech degree in information and communication technology from Indian Institute of Technology, Delhi, in 2008.
He received the PhD degree in electrical engineering from the University of Southern California (USC), Los Angeles, in
2014. He is currently a research staff member in the Watson Multimodal Group at IBM Watson. His research focuses on
...Show More
Kartik Audhkhasi received the BTech degree in electrical engineering
and the MTech degree in information and communication technology from Indian Institute of Technology, Delhi, in 2008.
He received the PhD degree in electrical engineering from the University of Southern California (USC), Los Angeles, in
2014. He is currently a research staff member in the Watson Multimodal Group at IBM Watson. His research focuses on
...View more
School of Interactive Computing, Georgia Institute of Technology, Atlanta, Georgia
Zachary Jacokes received the bachelor’s degree in psychology
from Emory University. His research interests include Autism Spectrum Disorders and traumatic brain injuries,
specifically regarding their effect on brain volumetrics, and neural connectivity. He is also interested in virtual
and augmented reality and their potential applications in treating Autism Spectrum Disorders (ASDs). He is currently a
data scientist ...Show More
Zachary Jacokes received the bachelor’s degree in psychology
from Emory University. His research interests include Autism Spectrum Disorders and traumatic brain injuries,
specifically regarding their effect on brain volumetrics, and neural connectivity. He is also interested in virtual
and augmented reality and their potential applications in treating Autism Spectrum Disorders (ASDs). He is currently a
data scientist ...View more
School of Interactive Computing, Georgia Institute of Technology, Atlanta, Georgia
Agata Rozga received the BA degree in psychology from the University
of California, Berkeley and the MA and PhD degrees in developmental psychology from the University of California, Los
Angeles. She completed a postdoctoral fellowship through the Center for Behavior Neuroscience at Georgia State
University. She is currently a senior research scientist in the School of Interactive Computing at Georgia Institute
of Tec...Show More
Agata Rozga received the BA degree in psychology from the University
of California, Berkeley and the MA and PhD degrees in developmental psychology from the University of California, Los
Angeles. She completed a postdoctoral fellowship through the Center for Behavior Neuroscience at Georgia State
University. She is currently a senior research scientist in the School of Interactive Computing at Georgia Institute
of Tec...View more
Department of Electrical Engineering, University of Southern California, Los Angeles, California
Shrikanth (Shri) Narayanan is an Andrew J. Viterbi professor of
engineering at the University of Southern California (USC), and holds appointments as a professor of electrical
engineering, computer science, linguistics, psychology, neuroscience and pediatrics and as the founding director of
the Ming Hsieh Institute. Prior to USC he was with AT&T Bell Labs and AT&T Research from 1995-2000. At USC he
directs the Signal ...Show More
Shrikanth (Shri) Narayanan is an Andrew J. Viterbi professor of
engineering at the University of Southern California (USC), and holds appointments as a professor of electrical
engineering, computer science, linguistics, psychology, neuroscience and pediatrics and as the founding director of
the Ming Hsieh Institute. Prior to USC he was with AT&T Bell Labs and AT&T Research from 1995-2000. At USC he
directs the Signal ...View more
Contents Department of Electrical Engineering, University of Southern California, Los Angeles, California
Rahul Gupta received the BTech degree in electrical engineering from
Indian Institute of Technology, Kharagpur, in 2010 and the PhD degree in electrical engineering from the University of
Southern California (USC), Los Angeles, in 2016. His research concerns development of machine learning algorithms with
application to human behavioral data. His dissertation work is on the development of computational methods for
modeling non-verbal communication in human interaction. He is the recipient of Info-USA exchange scholarship (2009),
Provost fellowship (2010-2014) and the Phi Beta Kappa alumni in Southern California scholarship (2015). He was part of
the team that won the INTERSPEECH-2013 and INTERSPEECH-2015 Computational Paralinguistics Challenges. He is a member
of the IEEE.
Rahul Gupta received the BTech degree in electrical engineering from
Indian Institute of Technology, Kharagpur, in 2010 and the PhD degree in electrical engineering from the University of
Southern California (USC), Los Angeles, in 2016. His research concerns development of machine learning algorithms with
application to human behavioral data. His dissertation work is on the development of computational methods for
modeling non-verbal communication in human interaction. He is the recipient of Info-USA exchange scholarship (2009),
Provost fellowship (2010-2014) and the Phi Beta Kappa alumni in Southern California scholarship (2015). He was part of
the team that won the INTERSPEECH-2013 and INTERSPEECH-2015 Computational Paralinguistics Challenges. He is a member
of the IEEE.View more
Watson Group, International Business Machines, Yorktown Heights, New York
Kartik Audhkhasi received the BTech degree in electrical engineering
and the MTech degree in information and communication technology from Indian Institute of Technology, Delhi, in 2008.
He received the PhD degree in electrical engineering from the University of Southern California (USC), Los Angeles, in
2014. He is currently a research staff member in the Watson Multimodal Group at IBM Watson. His research focuses on
automatic speech recognition, natural language processing, and machine learning. He was the recipient of the Annenberg
and IBM PhD fellowships. He was part of the team that won the INTERSPEECH-2013 Computational Paralinguistics
Challenge. He also received best paper and teaching assistant awards from the Electrical Engineering Department at USC
and was a 2012/13 Ming Hsieh Institute PhD Scholar. He is a member of the IEEE.
Kartik Audhkhasi received the BTech degree in electrical engineering
and the MTech degree in information and communication technology from Indian Institute of Technology, Delhi, in 2008.
He received the PhD degree in electrical engineering from the University of Southern California (USC), Los Angeles, in
2014. He is currently a research staff member in the Watson Multimodal Group at IBM Watson. His research focuses on
automatic speech recognition, natural language processing, and machine learning. He was the recipient of the Annenberg
and IBM PhD fellowships. He was part of the team that won the INTERSPEECH-2013 Computational Paralinguistics
Challenge. He also received best paper and teaching assistant awards from the Electrical Engineering Department at USC
and was a 2012/13 Ming Hsieh Institute PhD Scholar. He is a member of the IEEE.View more
School of Interactive Computing, Georgia Institute of Technology, Atlanta, Georgia
Zachary Jacokes received the bachelor’s degree in psychology
from Emory University. His research interests include Autism Spectrum Disorders and traumatic brain injuries,
specifically regarding their effect on brain volumetrics, and neural connectivity. He is also interested in virtual
and augmented reality and their potential applications in treating Autism Spectrum Disorders (ASDs). He is currently a
data scientist and programmer at the University of Southern California’s Lab of Neuro Imaging, where he manages
the database of ASD subjects for a project funded by the Autism Centers of Excellence. He is a member of the IEEE.
Zachary Jacokes received the bachelor’s degree in psychology
from Emory University. His research interests include Autism Spectrum Disorders and traumatic brain injuries,
specifically regarding their effect on brain volumetrics, and neural connectivity. He is also interested in virtual
and augmented reality and their potential applications in treating Autism Spectrum Disorders (ASDs). He is currently a
data scientist and programmer at the University of Southern California’s Lab of Neuro Imaging, where he manages
the database of ASD subjects for a project funded by the Autism Centers of Excellence. He is a member of the IEEE.View more
School of Interactive Computing, Georgia Institute of Technology, Atlanta, Georgia
Agata Rozga received the BA degree in psychology from the University
of California, Berkeley and the MA and PhD degrees in developmental psychology from the University of California, Los
Angeles. She completed a postdoctoral fellowship through the Center for Behavior Neuroscience at Georgia State
University. She is currently a senior research scientist in the School of Interactive Computing at Georgia Institute
of Technology, where she directs the Child Study Lab. A developmental psychologist and autism researcher by training,
she collaborates with computer scientists to develop novel computational tools and methods to objectively measure
behaviors relevant to studying typical and atypical development. Using these tools, she aims to shed new light on
early disruptions in social-communicative development in autism. He is a member of the IEEE.
Agata Rozga received the BA degree in psychology from the University
of California, Berkeley and the MA and PhD degrees in developmental psychology from the University of California, Los
Angeles. She completed a postdoctoral fellowship through the Center for Behavior Neuroscience at Georgia State
University. She is currently a senior research scientist in the School of Interactive Computing at Georgia Institute
of Technology, where she directs the Child Study Lab. A developmental psychologist and autism researcher by training,
she collaborates with computer scientists to develop novel computational tools and methods to objectively measure
behaviors relevant to studying typical and atypical development. Using these tools, she aims to shed new light on
early disruptions in social-communicative development in autism. He is a member of the IEEE.View more
Department of Electrical Engineering, University of Southern California, Los Angeles, California
Shrikanth (Shri) Narayanan is an Andrew J. Viterbi professor of
engineering at the University of Southern California (USC), and holds appointments as a professor of electrical
engineering, computer science, linguistics, psychology, neuroscience and pediatrics and as the founding director of
the Ming Hsieh Institute. Prior to USC he was with AT&T Bell Labs and AT&T Research from 1995-2000. At USC he
directs the Signal Analysis and Interpretation Laboratory (SAIL). His research focuses on human-centered signal and
information processing and systems modeling with an interdisciplinary emphasis on speech, audio, language, multimodal
and biomedical problems, and applications with direct societal relevance. [http://sail.usc.edu]. He is a
fellow of the Acoustical Society of America and the American Association for the Advancement of Science (AAAS) and a
member of Tau Beta Pi, Phi Kappa Phi, and Eta Kappa Nu. He is editor in chief for IEEE
Journal of Selected Topics in Signal Processing, an editor for the Computer Speech and Language
Journal and an associate editor for the IEEE Transactions on Affective Computing
, the APSIPA Transactions on Signal and Information Processing and the Journal of the
Acoustical Society of America. He was also previously an associate editor of the IEEE
Transactions of Speech and Audio Processing (2000-2004), the IEEE Signal
Processing Magazine (2005-2008), the IEEE Transactions on Multimedia
(2008-2011) and the IEEE Transactions on Signal and Information Processing Over Networks (2014-2015).
He is a recipient of a number of honors including Best Transactions Paper awards from the IEEE Signal Processing
Society in 2005 (with A. Potamianos) and in 2009 (with C. M. Lee) and selection as an IEEE Signal Processing Society
Distinguished Lecturer for 2010 and 2011 and ISCA Distinguished Lecturer for 2015-2016. Papers co-authored with his
students have won awards including the 2014 Ten-year Technical Impact award from ACM ICMI, Best Student Paper award at
ICASSP-2016, Interspeech 2015 Nativeness Detection Challenge, Interspeech 2014 Cognitive Load Challenge, Interspeech
2013 Social Signal Challenge, Interspeech 2012 Speaker Trait Challenge, Interspeech 2011 Speaker State Challenge,
InterSpeech 2009 Emotion Challenge, and other awards at IEEE DCOSS 2009, IEEE MMSP 2007, IEEE MMSP 2006, ICASSP 2005
and ICSLP 2002. He has published more than 700 papers and has been granted seventeen U.S. patents. He is a fellow of
the IEEE.
Shrikanth (Shri) Narayanan is an Andrew J. Viterbi professor of
engineering at the University of Southern California (USC), and holds appointments as a professor of electrical
engineering, computer science, linguistics, psychology, neuroscience and pediatrics and as the founding director of
the Ming Hsieh Institute. Prior to USC he was with AT&T Bell Labs and AT&T Research from 1995-2000. At USC he
directs the Signal Analysis and Interpretation Laboratory (SAIL). His research focuses on human-centered signal and
information processing and systems modeling with an interdisciplinary emphasis on speech, audio, language, multimodal
and biomedical problems, and applications with direct societal relevance. [http://sail.usc.edu]. He is a
fellow of the Acoustical Society of America and the American Association for the Advancement of Science (AAAS) and a
member of Tau Beta Pi, Phi Kappa Phi, and Eta Kappa Nu. He is editor in chief for IEEE
Journal of Selected Topics in Signal Processing, an editor for the Computer Speech and Language
Journal and an associate editor for the IEEE Transactions on Affective Computing
, the APSIPA Transactions on Signal and Information Processing and the Journal of the
Acoustical Society of America. He was also previously an associate editor of the IEEE
Transactions of Speech and Audio Processing (2000-2004), the IEEE Signal
Processing Magazine (2005-2008), the IEEE Transactions on Multimedia
(2008-2011) and the IEEE Transactions on Signal and Information Processing Over Networks (2014-2015).
He is a recipient of a number of honors including Best Transactions Paper awards from the IEEE Signal Processing
Society in 2005 (with A. Potamianos) and in 2009 (with C. M. Lee) and selection as an IEEE Signal Processing Society
Distinguished Lecturer for 2010 and 2011 and ISCA Distinguished Lecturer for 2015-2016. Papers co-authored with his
students have won awards including the 2014 Ten-year Technical Impact award from ACM ICMI, Best Student Paper award at
ICASSP-2016, Interspeech 2015 Nativeness Detection Challenge, Interspeech 2014 Cognitive Load Challenge, Interspeech
2013 Social Signal Challenge, Interspeech 2012 Speaker Trait Challenge, Interspeech 2011 Speaker State Challenge,
InterSpeech 2009 Emotion Challenge, and other awards at IEEE DCOSS 2009, IEEE MMSP 2007, IEEE MMSP 2006, ICASSP 2005
and ICSLP 2002. He has published more than 700 papers and has been granted seventeen U.S. patents. He is a fellow of
the IEEE.View more
