Reinforcement Learning based System-of-Systems Approach for UAV Swarms Behavioral Evolution

Advances in Unmanned Aerial Vehicle (UAV) technologies have enabled the development of biologically inspired swarms or fleets of UAVs that collaboratively achieve common mission objectives such as in surveillance and remote sensing applications. Typically, such swarms comprise of homogeneous UAVs that are developed using same techniques to serve the same purpose. However, the recent proliferation of UAV development by numerous research and commercial entities, using varied machine learning models to serve diverse purposes, has induced the need to rethink swarm development using heterogeneous UAVs that can adapt and achieve common objectives despite differences in their design and associated intelligence. In this paper, we present a novel approach to develop swarms of heterogeneous UAVs as System-of-Systems (SoS), and enabling behavioral evolution and adaptation of constituent UAVs through Reinforcement Learning (RL) algorithms. The proposed approach involves SoS-Constituent System Measures Of Effectiveness Relationship (SSMR) model that correlates the Measures Of Effectiveness (MOEs) for the SoS with the MOEs of the constituent system UAVs. By formulating the SSMR as the reward mechanism of RL, our approach incorporates an Intelligent-Behavior Evolution Agent with the necessary constraints to learn to maximize the SoS and system-level MOEs, while adapting itself to the evolution in the swarm SoS. The initial experimental results indicate that our approach adequately trains these independent agents to be used in heterogeneous swarms.