CNNPRE: A CNN-Based Protocol Reverse Engineering Method

Given the growth in computer networks and Internet usage, the traditional network environment has evolved into a more intricate system. Many applications utilize unknown communication protocols, for which the specification documentation is not available. The use of undocumented network protocols raises various security and management concerns. Protocol reverse engineering based on network traffic aims to infer the behavior and format of unknown network protocols. Clustering same-type messages or packets is a crucial initial step in correctly performing reverse engineering of protocol syntax or behavior. Therefore, this paper proposes a new method called CNNPRE, utilizing deep learning techniques to identify and group traffic message types. Our method employs network traffic and traffic features as input. Specifically, we use convolutional neural networks and deep transfer learning for feature extraction and message type identification and to tackle the challenge of unlabeled training data in the real world scenarios of protocol reverse engineering. The experimental results demonstrate that our proposed method works well and outperforms other methods for different protocols and achieves an average Homogeneity score of more than 0.87 on all datasets. This means that the method can identify message types according to the changing characteristics of messages and traffic features without the need for human expert intervention.