Hardware-in-the-loop and Digital Twin Enabled Autonomous Robotics-assisted Environment Inspection*

Empowered by the advanced 3D sensing, computer vision and AI algorithm, autonomous robotics provide an unprecedented possibility for close-up infrastructure environment inspection in an efficient and reliable fashion. Deep neural network (DNN) learning algorithms, pretrained on the large database can empower real-time object detection as well as fully autonomous, safe robotic navigation in unstructured environments while avoiding the potential obstacle. However, the development and deployment of the robots, inspection planning and operation procedures are still tedious and segmented with tremendous manual intervention during environmental inspection and anomaly monitoring. The proposed digital twin approach is able to provide a virtual representation model of the target environment either from a build-design or from 3D scanning of the real-world physical assets at high resolution in the Unity simulation environment, a transverse drone robot model and test its Robotics Operating System(ROS) autonomous navigation and obstacle avoidance software stack, and the hardware-in-the-loop test can thus be conducted for the flight control algorithm effectiveness and real-time object detection performance evaluation. The preliminary result shows that VGG16-UNet deep learning algorithm was able to use only a small amount of guidance and time from experienced inspection pilots to successfully identify the critical elements and defects and real-time navigate around the unstructured environment. The proposed digital twin framework and methodology is promising to be utilized for developing and testing fully autonomous inspection robots and its path planning and navigation and detection operation with greater cost- and time-efficiency.