AutoENP: An Auto Rating Pipeline for Expressing Needs via Pointing Protocol

Early screening for ASD (Autism Spectrum Disorder) is crucial and also challenging due to the limited medical resource. Expressing Needs with Pointing (ENP) is a low-cost yet effective protocol for early screening. However, the current methods need to manually trim video for analyzing ENP protocol, which is labour-intensive. Also, they detect discriminative signs with separately high-level clues (e.g., pose, object detection), but ignore the temporal action relationships between child and clinician, which usually leads to invalid detection. In contrast to previous approaches, we propose an Auto Rating Pipeline for Expressing Needs via Pointing Protocol, named AutoENP. Specifically, we introduce action segmentation into early screening, to capture temporal interaction relationships without manually intervention. To detect fine-grained hand motions, we fuse global, local and fine-grained features to fully understand the screening scene. Besides, we integrate focal loss and center loss to improve the detection accuracy for rare actions. To evaluate the proposed pipeline, we collected 22 ENP videos containing 7 actions with above 40,000 frames. Experimental results demonstrate that our model achieves 82.1% and 84.7% action accuracy for child and clinician, respectively. Moreover, 18 in 22 children’s ENP levels are reported correctly against the clinician’s diagnoses.