openLG: A Tunable and Efficient Open-source LSTM on GPUs

Long-Short-Term Memory (LSTM) neural networks have demonstrated exceptional proficiency in capturing both short-term and long-term dependencies within input sequences, rendering them invaluable across diverse applications, including DNA basecalling, speech recognition, reading comprehension, and scientific numerical forecasting. Despite their significance, the closed-source nature of state-of-the-art libraries, particularly the cuDNN LSTM inference kernel on NVIDIA GPUs, poses challenges in extending, optimizing, and understanding their internal workings.In this paper, we propose openLG, an open source framework to implement high performance LSTM kernel on basis of NVIDIA template kernel library CUTLASS. With flexible and transparent template configurations, developers are able to select the most suitable parameter combination for various scenarios. We formalize the LSTM computation as the time dependent and independent partitions, and then exploit efficient CUTLASS GEMM kernel for both parts. Evaluations conducted on NVIDIA GPUs demonstrate that openLG achieves up to 78% speedup on DeepBench benchmarks and an average speedup of 10% on realistic applications, effectively surpassing the performance of cuDNN.