Deep Learning Enhanced Feature Extraction of Potholes Using Vision and LiDAR Data for Road Maintenance

As the global population increases, so does the number of vehicles on our roads, which makes maintenance of the road infrastructure critical for safe and efficient transportation. A significant challenge in road maintenance is to address surface defects, such as potholes, which pose the risk of accidents and vehicle damage. This work proposes an automated solution to improve the detection and aid in the repair of potholes, thus reducing the reliance on manual inspections and reducing the overall maintenance time. Our methodology integrates LiDAR (Light Detection and Ranging) with RGB (Red, Green, and Blue) camera data to enhance depth information for accurate pothole characterisation. Geo-positioning using the GNSS (Global Navigation Satellite System) allows for precise mapping of detected potholes. An RGB image dataset created by aggregating publicly available pothole image datasets was used to train the object detection model YOLO (You Only Look Once) implemented in this work. Using this data, the models YOLOv5, YOLOv6, YOLOv7, and YOLOv8 were trained and their performance analysed. Remarkably, YOLOv5 showed the best implementation performance during the training phase, and it was lately selected for real time deployment. The data provided by the LiDAR sensor were used to compute the area, volume and depth of the detected pothole using the Convex Hull approaches. During deployment on Edinburgh City roads, our work was able to effectively detect and characterise 52 potholes of different volume and area. The implementation of this technology has the potential to significantly reduce inspection time, and our findings offer promising directions for future developments in automated road maintenance systems.