FAQ: A Flexible Accelerator for Q-Learning with Configurable Environment

Reinforcement Learning is an area of machine learning that is concerned with optimizing the behavior of an agent in an environment by maximizing cumulative rewards. This can be done with classical reinforcement learning algorithms such as Q-Learning and SARSA. This paper presents FAQ, a flexible FPGA-based accelerator for the Q-Learning algorithm. The architecture of the accelerator can be configured in multiple ways, like adjusting the bit width of Q-values or changing the number of pipeline stages. The evaluation shows that FAQ achieves 249% higher throughput than state-of-the-art FPGA implementations while decreasing DSP and BRAM utilization. Additionally, a software-configurable environment was implemented, and the whole system was tested on an Ultra96-V2 development board utilizing the PYNQ framework. Compared to a CPU implementation, FAQ is more than 13 times faster, including communication overhead caused by transferring the environment onto the FPGA and reading the resulting Q-table.