Optimal Robust Formation of Multi-Agent Systems as Adversarial Graphical Apprentice Games With Inverse Reinforcement Learning

This paper introduces a novel approach to solving robust optimal formation control problems in heterogeneous multi-agent systems (MASs) with disturbances. Our approach frames this problem as an adversarial graphical apprentice game problem, using the game concept and an inverse reinforcement learning (IRL) algorithm. This work innovates the derivation of unknown reward functions through demonstrations to achieve the desired formation. In addition, we define learner, estimator, and expert MASs as separate entities. This aim is achieved by introducing a model-based IRL algorithm with three stages of learning. After determining the optimal control of estimators in the first stage, the optimal control of learners is found in the second stage by using them. In the third stage, the reward functions of the estimator and learner MASs are updated in the inverse optimal control (IOC) stage simultaneously. Following that, we present an algorithm for model-free IRL that does not require knowledge of learner agents’ dynamics and can reconstruct reward functions after observing online trajectories. As subproblems, both proposed IRL algorithms address optimal control and IOC. We analyze these approaches for stability and convergence and demonstrate that state reward weights are non-unique. Simulation results show the effectiveness of the introduced algorithms for a group of unmanned aerial vehicles (UAVs).
Note to Practitioners
—Designing an optimal controller is challenging in MAS, especially where we only have access to the optimal reference path. The challenge is compounded if we do not know the agents’ dynamics and the appropriate reward function needed to achieve the desired path. This article proposes two IRL algorithms with the motivation to solve this problem. This study considers the optimal reference path for agents as expert MAS. Also, the MAS, we want to follow the desired path with the minor error, is represented as a learner MAS. The proposed approach is suitable for a large class of linear MAS since we have modeled the learner MAS with disturbance. Therefore, this method suits tracking, path-planning, and optimal formation problems in MASs. It requires minimal information but yields impressive results.