Why are My Flows Different? A Tutorial on Flow Exporters

Network flows build the basis of modern network data analysis by aggregating properties of network packets with common characteristics. A consistent and unambiguous definition of the network flow concept is an indispensable prerequisite and starting point for reproducible network research. However, in today’s practice, the flow output of distinct flow exporters, which is software to generate flows from observed network packets, varies substantially on identical network packet stream input. In this paper we present an in-depth comparison of different flow exporters and show how their outputs differ significantly. We argue that this substantially impairs reproducibility for traffic analysis research. We first present the detailed flow definition of the IP Flow Information eXport (IPFIX) standard including explanations and examples, analyze design and implementation of existing flow exporters, and explore the reasons why many projects and publications chose to implement their own flow exporters. Based on this analysis we highlight the main challenges in the flow exporting process and present a detailed tutorial on how to design and implement a flow exporter such that it yields consistent, reproducible output. Based on the tutorial’s theoretical analysis and lessons learned we present design and main concepts of a versatile, flexible, and open source flow exporting solution called
go-flows
that generates deterministic, reproducible network flows. Finally, we present a flow-by-flow comparison of the analyzed flow exporters’ output, explore the differences in terms of their generated flows, compare flow exporter performance, and conclude with guidelines on parameters that play a crucial role in improving the reproducibility of exported flows.