Evolutionary Deep Reinforcement Learning via Hybridizing Estimation-of-Distribution Algorithms with Policy Gradients

CEM-RL is a state-of-the-art evolutionary rein-forcement learning (ERL) framework to perform policy search for control problems with continuous action spaces. CEM-RL employs a cross-entropy method (CEM) variant, in essence a Gaussian Estimation-of-Distribution Algorithm (EDA), to model the distribution of high-performing control policies, which are typically parameterized by (deep) neural networks. For each iteration, new policies are generated from the learned distribution and are further improved by an actor-critic policy gradient procedure, in particular, Deep Deterministic Policy Gradient (DDPG) or Twin-Delayed DDPG (TD3). In this paper, we employ the Adapted Maximum-Likelihood Gaussian Model Iterated Density-Estimation Evolutionary Algorithm (AMaLGaM), which offers a wider range of customization options than CEM. Beside DDPG and TD3, we consider the recently-introduced Double Actors Regularized Critics (DARC), that can address the overestimation bias of DDPG and the underestimation bias of TD3, for the integration with AMaLGaM. Benchmark results on MuJoCo continuous locomotion tasks demonstrate the excellent performance