Reinforcement Learning-Based Optimal Formation Control for Multiple WMRs With Visual Servoing

In this paper, a reinforcement learning (RL) control method is developed for the formation control of multiple wheeled mobile robots (WMRs) with visual servoing. First, a multi-robot system model is constructed based on the kinematic models of mobile robots, the camera model, and the multiple-view geometry principles. The leader-follower structure is then applied to derive the distributed error system. Next, the error term is separated from the optimal performance index function, and the Bellman residual error is obtained based on the Hamilton-Jacobi-Bellman equation (HJBE). Subsequently, the gradient descent method is employed to design the weight update rate, which is implemented in an actor-critic neural network (NN) architecture. The proposed RL control method achieves formation tracking and performance optimization simultaneously, which previous approaches have not accomplished. Furthermore, under the Lyapunov stability theory, it is proven that the follower robots can track the leader in a predefined formation, and the tracking error converges ultimately. The simulation outcomes verify the effectiveness of the developed approach. Note to Practitioners—The primary motivation of this study is to establish a multiple WMRs system model based on visual servoing and to develop an RL control strategy. While visual servoing enhances the flexibility and accuracy of multi-robot systems, there remains a significant gap in research on implementing RL-based control for such models. To address this gap, the paper construct an error dynamics model that integrated multi-view geometry principles with robot kinematic constraints. Based on this model, the paper proposes a novel RL approach implemented through an actor-critic NN framework. Unlike conventional adaptive NNs, the parameters of the actor network are updated through the critic network, which enables more stable and efficient policy optimization by providing continuous performance feedback.