Evaluating routing asymmetry by passive flow measurements with spark

Routing asymmetry increases network efficiency, but also brings serious challenges to some use cases, such as policy enforcement, flow identification, malicious/anomaly detection, traffic shaping and limit etc. Routing asymmetry is an inevitable phenomenon. However, the question is that how is the specific asymmetric degree of network traffic. We classify and evaluate the conditions of network traffic actually routed asymmetry in the ISPs network that has a significant impact on the network security, measurement, model and management. For depicting asymmetric degree of the network packet in a flow and hosts behavior, we classify the routing asymmetry into flow asymmetry and IP asymmetry. In order to meet the high bandwidth, multiple observation points and passive measurement requirements, we build a scalable passive measurement system for online or offline evaluating routing asymmetry. The system adopts fast packet I/O technique to capture network traffic, and Spark to process the collected data. To prove its practicability, we deploy the system in a carrier grade network that has six data centers. We have found that (1) over 90% of TCP flows are asymmetric, and (2) over 70% of flows from full IP asymmetry that a same IP address traverse different links.