Scaling up Deep Reinforcement Learning for AI Using FPGAs

Deep Reinforcement Learning (DRL) combines Reinforcement Learning (RL) algorithms and Deep Learning (DL) to achieve remarkable advancements in AI. Specifically, RL trains one or more agents in an environment typically for estimation, optimization or control based tasks. DRL takes advantage of the high dimensional, complex, non-linear, universal function approximation capability of DL to extend RL. This allows DRL to support a very broad range of practical applications across many disciplines. This paper looks at the task of scaling up DRL algorithms in FPGAs to meet the rapidly growing demand of AI applications. FPGAs are largely underutilized for DRL applications but usage is expected to increase as demand for AI drives the need for high performance, complex DRL applications. The use of FPGAs for DRL represents a practical, field deployable, AI solution that offers several key advantages such as relatively low SWAP-C, scalable, fully reconfigurable, high throughput, and low latency. This paper begins with an overview and background of RL algorithms to provide context for the challenges of FPGA implementation. For example, similar to other types of DL architectures, DRL implementations must also implement back propagation in the FPGA in order to train the DL network and therefore allows an agent to learn. Design data as well as insight for scaling and implementing large DRL systems for AI applications using FPGAs is provided herein. This includes DRL challenges in the context of agents running on Multi-FPGA systems in a large-scale, high-throughput AI implementation. Finally, an example large DRL Multi-FPGA design is provided to illustrate the concepts in the paper using an AMD Versal device.