STMemAE: An Instance-Level Based Spatio-Temporal Memory Autoencoder for Unsupervised Vision-Based Seizure Detection

Electroencephalogram (EEG) is most favorable in epilepsy analysis, but suffered from inconvenient recording and ease in disturbation. Contrastively, the vision-based seizure detection is more feasible in real applications for 24/7 monitoring. However, most vision-based seizure detections follow the supervised training scheme, that is usually tedious and time-consuming in data annotations. With these regards, an effective instance-level based spatio-temporal memory autoencoder, called STMemAE, is proposed for unsupervised vision-based seizure detection in this paper. In STMemAE, YOLOv5 object detection algorithm is first applied to build an instance-level based memory AE since the frame-level based AE cannot well adapt to new scenarios. A convolutional encoder is then used to extract both the posture and motion features of the subjects. A two-stream memory net aiming to generate both spatial and temporal memory-augmented features is further developed to linearly combine the memory items. Finally, the outputs of two streams are delivered to decoder for detection. Moreover, the gradient loss is used together with the intensity loss in the training stage for better future frame generation on interictal data, which can also speedup the network training. The performance is evaluated on the video dataset recorded in Children's Hospital, Zhejiang University School of Medicine (CHZU), consisting video sequences of 15 childhood epilepsy patients. Results show that STMemAE can achieve 98.16% of Area Under Curve (AUC), and outperforms several popular deep learning models as well as many unsupervised vision-based seizure detection methods.