Multimodal People Re-Identification Using 3D Skeleton, Depth, and Color Information

Person re-identification aims at recognizing the same person across multiple cameras or over different video frames in varying viewpoints, environments, and lighting conditions. Appearance matching in these challenging situations can be complex. In the last few years, thanks to the availability of RGB-D data, appearance matching has been addressed by using 3D body models in addition to the traditional 2D features commonly used in earlier methods. This paper proposes a novel method for person re-identification that builds a color-based signature adapted to the 3D body parts observed by multiple cameras. To be sure of comparing color features coming from the same part of the body, independently from the people's posture and shape, a spatial appearance signature has been built to capture more information from regions likely to contain more informative color content. The proposed method consists of a number of steps. In a preliminary phase, the point cloud related to the person to be recognized is opportunely registered to a standard reference system to keep viewpoint invariance. Then, the skeleton joints of the person enable the clustering of the 3D body reconstruction in different portions. Each part of the person's body can be analyzed using 3D adaptive partition grids of different sizes, and a color-based descriptor can be extracted from each cell to compose the person's signature. The signature is then compared with the other ones in the reference database to perform the person's re-identification. The robustness and effectiveness of the proposed solution have been extensively proven on three publicly available datasets (BIWI-RGBD-ID, KinectREID, RGBD-ID). An improvement of the current state of the art is obtained, achieving recognition rates of 99.6% (BIWI-RGBD-ID), 68.9% (KinectREID), and 92.8% (RGBD-ID), respectively.