FastAct: A Lightweight Actor Compression Framework for Fast Policy Learning

Deep reinforcement learning outperforms humans in a variety of complex decision-making tasks. However, training an effective policy requires millions of interactions between the agent and the target environment, making the learning process time consuming and resource intensive. Inspired by neural network compression, we propose a lightweight policy learning framework, FastAct, to accelerate policy training by speeding up Actor inference. Specifically, FastAct accelerates the Actor by dynamically using multiple compression techniques to quickly generate experience data for policy training. Compared to the traditional framework, the FastAct adds two modules, Compressor and Scheduler. The Compressor integrates multiple compression algorithms through the chain-of-responsibility design pattern and aims to maximize the inference speed of the Actors. To ensure the quality of the experience data, we develop the Scheduler to dynamically adjust the compression algorithm according to the statistical information of the Actor. In addition, FastAct applies the off-policy value estimator V-trace to correct the value function of the target policy. We implemented FastAct based on IMPALA and conducted extensive experiments on Atari. Compared with IMPALA, FastAct speeds up the Actor by 2.0x to 4.1x. And FastAct reduces training time by 29.8% to 40.3% while maintaining a similar result.