Lidar-Based Gait Analysis and Activity Recognition in a 4D Surveillance System

This paper presents new approaches for gait and activity analysis based on data streams of a rotating multibeam (RMB) Lidar sensor. The proposed algorithms are embedded into an integrated 4D vision and visualization system, which is able to analyze and interactively display real scenarios in natural outdoor environments with walking pedestrians. The main focus of the investigations is gait-based person reidentification during tracking and recognition of specific activity patterns, such as bending, waving, making phone calls, and checking the time looking at wristwatches. The descriptors for training and recognition are observed and extracted from realistic outdoor surveillance scenarios, where multiple pedestrians are walking in the field of interest following possibly intersecting trajectories; thus, the observations might often be affected by occlusions or background noise. Since there is no public database available for such scenarios, we created and published a new Lidar-based outdoor gait and activity data set on our website that contains point cloud sequences of 28 different persons extracted and aggregated from 35-min-long measurements. The presented results confirm that both efficient gait-based identification and activity recognition are achievable in the sparse point clouds of a single RMB Lidar sensor. After extracting the people trajectories, we synthesized a free-viewpoint video, in which moving avatar models follow the trajectories of the observed pedestrians in real time, ensuring that the leg movements of the animated avatars are synchronized with the real gait cycles observed in the Lidar stream.