Reinforcement Learning-Based Switching Controller for a Milliscale Robot in a Constrained Environment

This work presents a reinforcement learning-based switching control mechanism to autonomously move a ferromagnetic object (representing a milliscale robot) around obstacles within a constrained environment in the presence of disturbances. This mechanism can be used to navigate objects (e.g., capsule endoscopy, swarms of drug particles) through complex environments when active control is a necessity but where direct manipulation can be hazardous. The proposed control scheme consists of a switching control architecture implemented by two sub-controllers. The first sub-controller is designed to employ the robot’s inverse kinematic solutions to do an environment search for the to-be-carried ferromagnetic particle while being robust to disturbances. The second sub-controller uses a customized rainbow algorithm to control a robotic arm, i.e., the UR5 robot, to carry a ferromagnetic particle to a desired position through a constrained environment. For the customized Rainbow algorithm, Quantile Huber loss from the Implicit Quantile Networks (IQN) algorithm and ResNet are employed. The proposed controller is first trained and tested in a real-time physics simulation engine (PyBullet). Afterward, the trained controller is transferred to a UR5 robot to remotely transport a ferromagnetic particle in a real-world scenario to demonstrate the applicability of the proposed approach. The experimental results on the UR5 robot show an average success rate of 98.86% over 30 episodes for randomly generated trajectories, demonstrating the viability of the proposed approach for real-life applications. In addition, two classical path finding approaches, Attractor Dynamics and the execution extended Rapidly-Exploring Random Trees (ERRT), are also investigated and compared to the RL-based method. The proposed RL-based algorithm is shown to achieve performance comparable to that of the tested classical path planners whilst being more robust to deploy in dynamical environments. Note to Practitioners —Deep reinforcement learning methods have been widely applied in computer games and simulations. However, employing these algorithms for practical, real-world applications such as robotics becomes challenging due to the difficulty of obtaining training samples. This paper predominantly focuses on bridging the gap between simulations and the real-world implementation of a reinforcement learning algorithm for a robotic application in the context of miniaturized drug delivery robots and robotic capsule endoscopes. This paper presents the derivation and experimental validation of a reinforcement learning-based algorithm for controlling a magnetically-actuated small-scale robot within a simplified model of the large intestine in the presence of disturbances. We demonstrate the possibility of training a high-fidelity reinforcement learning algorithm fully within a simulated environment before deploying it as-is in a real-world scenario by carrying out different experiments and simulations. Implementing the presented control framework complements a large body of this work, and the results offer a feasibility study of using reinforcement learning algorithms in practice.