Object Classification and Semantic Mapping

Rescue forces require quick and essential information about the unknown situation in order to achieve an efficient mission planning. For this purpose, micro air vehicles can provide this kind of information without risking harm of the task force members. Hence, the objective of this paper is to provide semantic object information of the explored area. In addition, it is of interest to know where the object is positioned. The paper deals with various challenges occurring in this application: Different observations of the same object during flight, close objects of the same and different classes, and no prior knowledge about the number of objects. The output of a fine-tuned region-based convolution neural network (R-CNN), i.e. detected object class, score and bounding box, is used for tracking over subsequent images. The corresponding features along with the navigation solution are input of a multi-view triangulation step. The resulting point cloud is further processed so that clusters are identified. Then, the bounding boxes are associated with the qualifying cluster. Based on that, the sizes of the found objects are estimated by introducing an outlier rejection strategy. The holistic approach is evaluated by obtaining real data in a typical office environment. The results show a comprehensive semantic map that includes multi-class objects of different sizes and shapes.