A Scalable Low-Power Reconfigurable Accelerator for Action-Dependent Heuristic Dynamic Programming

Adaptive dynamic programming (ADP) is an effective algorithm that has been successfully deployed in various control tasks. For many emerging applications where power consumption is a major design consideration, the conventional way of implementing ADP as software executing on a general-purpose processor is not sufficient. This paper proposes a scalable and low-power hardware architecture for implementing one of the most popular forms of ADP called action-dependent heuristic dynamic programming. Different from most machine-learning accelerators that mainly focus on the inference operation, the proposed architecture is also designed for energy-efficient learning, considering the highly iterative and interactive nature of the ADP algorithm. In addition, a virtual update technique is proposed to speed up the computation and to improve the energy efficiency of the accelerators. Two design examples are presented to demonstrate the proposed algorithm and architecture. Compared with the software approach running on a general-purpose processor, the accelerator operating at 175 MHz achieves 270 times improvement in computational time while consuming merely 25 mW power. Furthermore, it is demonstrated that the proposed virtual update algorithm can effectively boost the energy efficiency of the accelerator. Improvements up to 1.64 times are observed in the benchmark tasks employed.