Adaptive Modulation for Underwater Acoustic Communications Based on Reinforcement Learning

Extended propagation delay is the major challenge in implementing adaptive modulation schemes in underwater acoustic communications. Due to the long propagation delay, the channel state information is often outdated when received at the transmitter, thus limiting the gain from the feedback or even causing communication failures. The adaptive modulation problem can be formulated as a Markov decision process, which can be solved by reinforcement learning methods. In this paper, an adaptive modulation strategy was developed based on reinforcement learning, specifically the Dyna-Q algorithm, to improve the communication throughput in the underwater acoustic environment. The acoustic communication channel was characterized by a single parameter, the effective signal-to-noise ratio (ESNR), which was referred to as the channel state. The Dyna-Q algorithm undertook two roles: predicting the channel state and calculating communication throughput based on the real and simulated experiences of data communications for each modulation order at different channel states. With the predicted channel state and the calculated throughputs, the Dyna-Q algorithm chose the modulation order based on the returned channel state from the receiver, with a goal to maximize the long-term reward, i.e., communication throughput. Such Dyna-Q based adaptive modulation strategy was applied to data communications with an autonomous underwater vehicle in the simulated acoustic communication channel. The performance of the adaptive modulation system was examined when the vehicle traveled at two different tracks. An acoustic parabolic equation model, Monterey-Miami Parabolic Equation (MMPE), with historical sound speed profile measurements was used to generate acoustic impulse responses. Our simulations showed that the Dyna-Q algorithm achieved higher throughputs than the direct feedback of the ESNR.