Unknown application layer protocol recognition method based on deep clustering

In recent years, many unknown protocols are constantly emerging, and they bring severe challenges to network security and network management. Existing unknown protocol recognition methods suffer from weak feature extraction ability, and they cannot mine the discriminating features of the protocol data thoroughly. To address the issue, we propose an unknown application layer protocol recognition method based on deep clustering. Deep clustering which consists of the deep neural network and the clustering algorithm can automatically extract the features of the input and cluster the data based on the extracted features. Compared with the traditional clustering methods, deep clustering boasts of higher clustering accuracy. The proposed method utilizes network-in-network (NIN), channel attention, spatial attention and Bidirectional Long Short-term memory (BLSTM) to construct an autoencoder to extract the spatial-temporal features of the protocol data, and utilizes the unsupervised clustering algorithm to recognize the unknown protocols based on the features. The method firstly extracts the application layer protocol data from the network traffic and transforms the data into one-dimensional matrix. Secondly, the autoencoder is pretrained, and the protocol data is compressed into low dimensional latent space by the autoencoder and the initial clustering is performed with K-Means. Finally, the clustering loss is calculated and the classification model is optimized according to the clustering loss. The classification results can be obtained when the classification model is optimal. Compared with the existing unknown protocol recognition methods, the proposed method utilizes deep clustering to cluster the unknown protocols, and it can mine the key features of the protocol data and recognize the unknown protocols accurately. Experimental results show that the proposed method can effectively recognize the unknown protocols, and its performance is better than other methods.