An FPGA Implementation of GCN with Sparse Adjacency Matrix

Deep learning has revolutionized many machine learning tasks in recent years, ranging from image classification and video processing to speech recognition and natural language understanding. The data in these tasks are typically represented in the Euclidean space. However, there is an increasing number of applications where data are generated from non-Euclidean domains and are represented as graphs with complex relationships and interdependency between objects. The complexity of graph data has approaches for graph data have emerged. Graph convolution networks (GCNs) in particular, try to replicate the success of CNN in graph data by defining graph convolutions via graph spectral theory or spatial locality. This paper presents a new GCN accelerator for the graph convolution layers of ST-GCN [1], which is successfully applied in action recognition. The accelerator breaks down the graph convolution into convolution and matrix multiplication with adjacency matrix. To optimize the power efficiency, the dataflow of the convolution is designed properly and a sparse matrix-vector multiplication is proposed to make use of the sparsity of the adjacency matrix. The accelerator is implemented on NSA.241 accelerator and can reach its peak performance of 46.0GOP/s under 188MHz with about 220 DSP, which achieves high DSP efficiency.