2D-3D semantic segmentation using cardinality as higher-order loss

Multi-modal scene analysis is a growing field of importance as additional sensors, such as 3D LIDAR, is becoming a common complement to image capturing systems. However, while additional sensory data potentially can make the analysis more accurate, it also comes with a host of associated issues. For example, inconsistencies in the data between sensors resulting from, e.g., misalignment, moving objects, or parallax effects, can severely affect the performance. Additionally, real-world scenes tend to have an inherent imbalance in the number of items of each class which typically suppresses the performance of infrequent classes. In this paper, we address those two issues specifically by a) using a cardinality loss function designed to target inconsistencies at training time, and b) devising an average per class loss function addressing the imbalance issue.