Dual Optimization-Based Distributed Tracking Control Under Completely Unknown Dynamics

Unlike existing results on output tracking control in multi-agent systems, which mainly focus on the relative state among agents, in some cases the state of the system may not be available or measurable. In this paper, we investigate the model-free learning-based distributed optimal tracking control for heterogeneous multi-agent systems with dynamic output feedback under a switching reinforcement learning algorithm. First, a relative output-based distributed observer is developed without exchanging state information, which can significantly reduce the interaction load and broaden the range of applications. Then, a distributed feedback-feedforward controller is proposed, where the optimal feedback and feedforward gain matrices can be learned online by solving two optimization problems, employing policy iteration-based reinforcement learning instead of relying on the leader’s state as in existing studies. Subsequently, the policy iteration algorithm (PI) is modified into the value iteration (VI) learning algorithm, which can relax the requirement for an initial stabilizing control policy and does not depend on the known dynamical model. Additionally, a switching reinforcement learning algorithm is put forward by fully integrating the merits of the previously mentioned methods. The new algorithm not only overcomes the initial stabilizing assumption, but also ensures the convergence of the algorithm in a model-free fashion. Finally, a simulation example is provided to illustrate the theoretical analysis.
Note to Practitioners
—This paper investigates the model-free learning-based distributed optimal tracking control for heterogeneous multi-agent systems with dynamic output feedback under a switched reinforcement learning algorithm. The proposed distributed cooperative control algorithms can be applied to multiple ground vehicles, air vehicles, and underwater vehicles. Unlike existing results on output tracking control, the obtained results rely on accurate system dynamics and ignore the transient performance, which makes the designed controller far from optimal in potential applications. Moreover, the optimal learning algorithm strictly relies on the initial stabilizing control policy related to accurate system dynamics, and it may be helpless when the system model is completely unknown. To overcome those issues, we developed a new data-driven algorithm or data-based model-free reinforcement learning algorithm to study distributed output tracking control with relative output information by collecting system data instead of using accurate system dynamics. The new algorithm not only overcomes the initial stabilizing assumption but also ensures the algorithm convergence in a model-free fashion. Potential applications of the proposed control algorithms include cooperative formation control, and secondary control of microgrids.