3D spatial condition sharing system based on sensor fusion of LiDAR and depth camera

With the recent spread of COVID-19, online communication methods such as web conferencing are attracting attention. However, the existing methods have a problem that it is difficult to get the same realistic experience as in real world. Existing studies propose online communication methods using avatars in a virtual reality space, but cannot completely solve the problem because the avatar’s appearance is not the same as in the real world and it is difficult for the avatars to reflect the user’s real-world behavior. Especially, when an elderly people living alone communicates with family members living far away, it is ideal to generate a three-dimensional virtual communication space that is close to the real world and to provide an experience as if the users are living in the same space. In order to observe the three-dimensional shape of the structures in the real world, a 3D LiDAR (Light Detection and Ranging) and a depth camera are often utilized. However, these devices have problems that the LiDAR can observe a wide area of the real world but cannot acquire color information, and that the depth camera can acquire both depth and color information but can only accurately observe the limited area. This study proposes a combination of the advantages of the 3D LiDAR and the depth camera to be able to generate a realistic communication space. In the proposed system, the 3D LiDAR is utilized to observe the shape of the structures in the room and both location and behavior of the people living in the room as point cloud data. In addition, the depth camera obtains accurate shape and color of the 3D structures in the room, and the color information is reflected to the point cloud data by applying the machine learning technologies that extract only targeted people. Therefore, our study can solve the existing problem by participating the constructed 3D space through a HMD (Head Mounted Display), the family members living far away from each other can get the realistic experience as if they were living together.