Deep Reinforcement Learning for Adaptive Exploration of Unknown Environments

Autonomous exploration is an essential task for unmanned aerial vehicles (UAVs) operating in unknown environments. Often, UAVs on these missions must first build a map of the environment via pure exploration and subsequently use (i.e. exploit) the generated map for downstream navigation tasks. Accomplishing these navigation tasks in two separate steps is not always possible and can potentially be disadvantageous for UAVs deployed in outdoor and dynamically changing environments. Current exploration approaches typically use a priori human-generated maps or heuristics such as frontier-based exploration. Other approaches use learning but focus only on learning policies for specific tasks and use sample inefficient random exploration or make impractical assumptions about full map availability. In this paper, we develop an adaptive exploration approach that allows for a trade off between exploration and exploitation in a single step using Deep Reinforcement Learning (DRL). We specifically focus on UAVs searching for areas of interest (AoIs) in an unknown environment. The proposed approach uses a map segmentation technique to decompose the environment map into smaller, tractable maps. DDQN and A2C algorithms are extended with a stack of LSTM layers and trained to generate optimal policies for the exploration and exploitation tasks, respectively. Then, an information gain function is repeatedly computed to determine the optimal trade-off between them. We tested our approach in 3 different tasks against 4 baselines. The results demonstrate that our proposed approach is capable of navigating through randomly generated environments and covering more AoI in less time compared to the baselines.