Reinforcement Learning Based Persistent Surveillance on Graphs Under Energy Constraints

In this work, we consider the problem of optimally routing a single agent on a graph for persistent surveillance and reporting under energy constraints. We aim to minimize a combination of weighted graph idleness, based on node priorities, and the time elapsed since the agent's last report. This requires repeatedly solving ‘similar’ computationally challenging optimization problems. To this end, we adopt a rolling horizon approach and use a deep Q-network (D3QN) based algorithm to learn policies that, given the graph state and the agent's location, decide the agent's next node to visit, while guaranteeing the satisfaction of energy constraints. Through exhaustive simulations and comparisons against an integer programming (IP) solver, we demonstrate near-optimality of our approach. Our method also provides several orders of magnitude reduction in computation time over the IP solver. We also compare our approach with five other frameworks, namely, Q learning based Black Box Learning Agent (BBLA), Entropy Maximized Patrolling (EMP), Travelling salesman problem (TSP) heuristic, Concurrent Bayesian Learning Strat-egy (CBLS), a graph attention (GAT) based learning method.