Understanding the Network Traffic Constraints for Deep Packet Inspection by Passive Measurement

The Deep Packet Inspection (DPI) system examines each captured network packet to find malicious events. The quality of network traffic set an upper limit of DPI system's functionality, such as traffic integrity and asymmetric routing. To better understand these constraints on a DPI system, we setup a series of indicators to quantify these factors, these indicators can be classified as basic information (e.g. average packet sizes), link stability (e.g. out of order packets number), connection integrity (e.g. missing packets number) and asymmetric routing (e.g. one-way flow). There are two challenges in measuring these indicators on real network traffic. The first is how to measure these indicators on high-speed traffic with limited resources. The second is how to track TCP flows across multiple inspection points. We tackle this problem with a scalable passive measurement system, which adopts fast packet I/O technique to capture network traffic, and Spark to process the collected data. To prove its practicability, we deploy the system in a carrier grade network that has six data centers. We have found that 1) over 90% of TCP SYN packets have no subsequent data packet, 2) over 90% of TCP flows are asymmetric, unordered or retransmitted, and 3) over 80% of TCP flow's round trip time are less than 400ms.