Deep Batch-Normalized LSTM networks with Auxiliary classifier for Skeleton based Action Recognition

Recently, Recurrent Neural Network (RNN) based approach presents good performance in skeleton based action recognition. Utilizing deep layers of RNN or Long-Short Term Memory (LSTM) has its on weakness when handling long-sequence data because of vanishing or exploding gradients through time. Batch Normalized LSTM (BN-LSTM) is able to give a solution for the problem, with the merit of converging faster in training. In contrast, when deeply layered, BN-LSTM structure shows slow convergence and worse accuracy. In this work, we analyze deep-layered BN-LSTM shows slower convergence in early training phase in training scheme and, finally we propose a deep BN-LSTM structure with auxiliary classifier that is able to converge faster and gives better results at skeleton based action recognition problem. Some experiment are conducted with Penn Action dataset and our own Computer Assembling Video dataset, we verified our proposal shows better results in skeleton based action recognition.