Deep Packet Inspection at Scale: Search Optimization Through Locality-Sensitive Hashing

Deep packet inspection is a primary tool for security specialists, surveillance analysts, and network engineers to lawfully intercept and analyze network traffic. In order to process this data or select streams of interest from the large amount of data flowing in today’s internet, solutions must be capable of identifying network traffic as quickly and accurately as possible. The ever-increasing diversity of data as well as sheer size has rendered the current regular expression matching and filtering solutions ineffective. We propose locality-sensitive hash embedding techniques Alpine and Palm for packet analysis. The fixed size of hashes as well as the adaptability of distance measures is proven to address the network traffic classification problem in our experiments and improves scalability over current state-of-the-art, automata-based search engines. In this paper, we analyze the system’s ability to classify network traffic by many data layer protocols and traffic types with over 99% accuracy. The model is also proven effective in areas where the regular expressions are inapplicable, such as traffic profiling. Finally, we provide real benchmarks of the system’s ability to scale to large signature and hash sets with much improved performance, demonstrating real-world applicability and generalizability of locality-sensitive hashing to deep packet inspection technology.