Simultaneous Transparent and Non-Transparent Object Segmentation With Multispectral Scenes

For an autonomous mobile system such as an autonomous robot that moves throughout a city, semantic segmentation is important. Performing semantic segmentation under diverse conditions, in turn, requires 1) a robust ability to recognize objects in low-visibility environments, such as at night and 2) the ability to recognize objects that transmit visible light, such as glass and acrylic used in doors and windows. To satisfy these requirements, using RGB images and infrared images simultaneously is considered effective. Visibility and infrared transmission characteristics are different for different objects; therefore, merely entering them into the conventional semantic segmentation framework is not applicable. For example, when a pedestrian is present behind a glass, the visible image captures the pedestrian rather than the glass and the infrared image captures the glass. In this research, we propose a new semantic segmentation method having a three-stream structure, focusing on the difference in the transmission characteristics. This method extracts not only valid features for ordinary non-transparent objects but also features effective for the recognition of transparent objects by utilizing differences in objects to be imaged owing to transmission characteristics. Furthermore, we constructed a new dataset called “coaxials” for the visible and infrared coaxial dataset and demonstrated that we can obtain better segmentation performance compared with the conventional method.