DQELR: An Adaptive Deep Q-Network-Based Energy- and Latency-Aware Routing Protocol Design for Underwater Acoustic Sensor Networks

Underwater acoustic sensor networks (UASNs) have become a popular research topic, with research challenges focused on underwater communication techniques. By incorporating long end-to-end latency, high energy consumption and dynamic network topology in UASNs, many intelligent routing protocols have been proposed to solve the problem. However, shortcomings still exist, and comprehensive routing protocols are urgently needed. In this paper, we propose an adaptive Deep Q-Network-based energy- and latency-aware routing protocol (DQELR) to prolong network lifetimes in UASNs. In the DQELR, a Deep Q-Network algorithm with both off-policy and on-policy methods is adopted to make globally optimal routing decisions. Based on both the energy and depth states of nodes at different communication stages, nodes with the maximum Q-value can be selected as forwarders adaptively considering both energy and latency. A hybrid of the broadcast and unicast communication mechanisms is also designed to reduce network overhead. In addition, network topology changes can be addressed through an on-policy method that makes a new routing decision when the current route becomes corrupted. With less energy consumption and strict latency limitations, the DQELR can prolong network lifetimes in UASNs. Simulation results show that the DQELR can achieve a superior network lifetime with better latency and energy efficiency performances relative to other general schemes applied in UASNs.