Point, Segment, and Inspect: Leveraging Promptable Segmentation Models for Semi-Autonomous Aerial Inspection

Operating unmanned aerial vehicles (UAVs) for assets inspections poses distinct challenges encompassing the need to maintain a safe distance from the inspection area, ensure correct orientation towards the inspected surface, and achieve comprehensive coverage of the entire surface. Achieving these tasks is inherently complex and stressful. Therefore, a novel approach that seeks to enhance the piloting experience by harnessing the latest advancements in segmentation models, such as Segment Anything Model (SAM), is proposed. These models, thanks to their prompting capabilities, allow seamless communication between the operator and the UAV, opening up the possibility of defining intricate inspection regions through simple interactions. Within this approach, decision-making authority remains with the operator, while the UAV takes on the demanding task of segmenting the designated area and devising an appropriate traversal plan. Throughout this process, the operator’s situational awareness is heightened through visual cues overlaid on the camera stream and a 3D panel presenting information of the drone position and spatially sensed data. This teleoperation framework allows the operator to maintain continuous control of the ongoing operation through a simplified interface. The paper delineates both the system and the methodology employed, showcasing the effectiveness of integrating segmentation models into the decision-making workflow. The validity of the proposed framework is established through testing within a photorealistic UAV simulator along with real experiments in a controlled laboratory environment.