A Rapid Situational Awareness Development Framework for Heterogeneous Manned-Unmanned Teams

This paper presents a robust framework for configuring and deploying a heterogeneous team of smart unmanned systems and human agents in dynamic and un-modeled environments to rapidly build mission critical situational awareness with selective details of potential areas of interest, especially focusing on minimized cognitive loading of the human agents. Five key components, namely control, communication, artificial intelligence (AI), platform, and visualization, merge seamlessly into a holistic framework to deliver this rapid situational awareness development capability to the heterogeneous manned unmanned team (MUM-T). In this framework, the overall control is seen as a combination of agent level control and mission level control. A common software, Robot Operating System (ROS), is used to establish communication, and consequently consensus, among the heterogeneous swarm of unmanned systems. These unmanned platforms are customized with co-processing hardware that can execute advanced artificial intelligence machine learning (AI/ML) modules to not only deliver stable and cooperative performance of these unmanned platforms in the swarm but also support human-centric human robot interaction (HRI). Finally, to reduce the cognitive burden on the human agents, a triaged visualization scheme, enabled through mixed reality (MR) technology, is implemented. This paper presents a preliminary proof of concept study for the presented hybrid map (i.e. 2D mapping with 3D detailing) construction framework, tested with a heterogeneous swarm of unmanned aerial vehicles (UAVs) of varying capabilities, teamed with a human operator.