Recognition Of Atypical Behavior In Autism Diagnosis From Video Using Pose Estimation Over Time

Autism spectrum disorder (ASD), similar to many other developmental or behavioral conditions, is difficult to be precisely diagnosed. This difficulty increases when the subjects are young children due to the huge overlap between the ASD symptoms and typical behaviors of young children. Therefore, it is important to develop reliable methods that could help distinguish ASD from normal behaviors of children. In this paper, we implemented a computer vision based automatic ASD prediction approach to detect autistic characteristics in a video dataset recorded from a mix of children with and without ASD. Our target dataset contains 555 videos, out of which 8349 episodes (each approximately 10 seconds) are derived. Each episode is labeled as an atypical or typical behavior by medical experts. We first estimate children pose in each video frame by re-training a state-of-the-art human pose estimator on our manually annotated children pose dataset. Particle filter interpolation is then applied on the output of the pose estimator to predict the locations of missing body keypoints. For each episode, we calculate the children motion pattern defined as the trajectory of their keypoints over time by temporally encoding the estimated pose maps. Finally, a binary classification network is trained on the pose motion representations to discriminate between typical and atypical behaviors. We were able to achieve a classification accuracy of 72.4% (precision =0.72 and recall =0.92) on our test dataset.