Using face and object detection to quantify looks during social interactions

Quantifying gaze is important in various realms, such as evaluating atypical social looking behavior in autism spectrum disorder. This paper reports on a system that uses eye-tracking glasses and object/face detection to quantify looks. The algorithms use Viola-Jones face detection with feature point tracking and Faster-RCNN object detection trained for three objects, followed by a runlength algorithm to declare the start and end of looks. Results are presented in terms of bounding box overlap and accuracy of looks compared to a manual ground truth. The system can be useful for quantifying gaze behavior during dynamic social interactions.