Terrain Preview Detection and Classification in Unstructured Scenes Based on Vision and Laser Fusion With Deep Learning

The safety and ride comfort of the emergency rescue vehicle can be improved greatly when the prior knowledge of the terrain in front of the vehicle is used as the active suspension input. In this paper, we present a robust and precise terrain preview detection and classification system based on vision and laser fusion with deep learning. First, a terrain classification method based on ResNet50 and transfer learning fusion is proposed to identify the terrain type in real time and improve the terrain classification accuracy. Second, a robust and precise terrain preview mapping model is developed to integrate the LiDAR with IMU data based on tightly coupled fusion and overcome the accumulation error difficulties caused by multisensor fusion. Third, introducing a decision-level fusion strategy with LiDAR and vision fusion enables the terrain classification results and three-dimensional terrain geometry information to be aligned in space and time, thereby obtaining effective terrain elevation information. As a result, the problem of inaccurate terrain elevation information can be solved. Based on the terrain detection and classification system framework, combined with the improved strategies above, the terrain classification accuracy of the training dataset and test dataset reaches 98% and 83%, respectively. The terrain mapping accuracy achieves approximately 4–5 cm in the large scene. Hence, our method effectively solves poor terrain elevation accuracy caused by the accumulated error of multisensor fusion. The experimental results highlight that the accuracy and robustness of the proposed terrain preview detection and classification algorithm in dealing with rapid vehicle rotation results are better than those of other current methods.