A Near-Optimal Solution to a Class of Deep Structured Teams with Nonlinear Dynamics

Classical control offers reliable, optimal control strategies in multi-agent settings to sequential decision-making problems. However, these methods usually rely on strong assumptions on the system dynamics. In contrast, model-free control provides the flexibility to deal with unmodeled system dynamics to perform complex tasks. In this work, we study the connection between the two methods in multi-agent systems. We focus on deep structured teams where the agent’s evolution is a known linear function of each agent’s state and the linear regression of all agents’ states and actions plus an unknown nonlinear term with a bounded Lipschitz constant. Furthermore, the cost is considered to be quadratic for the states and actions of all the agents. We prove the existence of a near-optimal solution in the convex vicinity of initialized controllers obtained from model-based LQR methods. We show these initialized control strategies are derived by solving an Algebraic Riccati Equation (ARE), obtained by neglecting the nonlinear terms. Finally, we provide convergence guarantees to the optimal solution using a derivative-free policy gradient approach. Simulations confirm the validity of the analytical results.