Towards holistic autonomous obstacle detection in railways by complementing of on-board vision with UAV-based object localization

This paper presents the two sub-systems of the first holistic system for autonomous obstacle detection (OD) in railways, the on-board vision system and unmanned aerial vehicle (UAV) vision system for object localization (OL) on and near the rail tracks. The main goal of such a holistic system is to enable long-range detection of obstacles on the rail tracks ahead of the train where the UAV-based OL system complements the on-board system in detecting obstacles in the areas that are not visible by the on-board system such as curves. The deep learning (DL)-based object detection and distance estimation in thermal camera unit of the on-board system is presented, as well as the OL based on UAV camera image. The experimental results achieved in a real-world railway experimental scenario that includes obstacles visible only by the on-board thermal camera, only by UAV camera as well as obstacles in the Field of View (FoV) of both systems are presented. These preliminary results show the high potential of developing holistic system where the final decision on OD would be autonomously made and consequently possible actions for the train control would be suggested based on the OD outputs of individual systems having different rail tracks sections in their FoV.