A Probabilistic Counting Framework for Distributed Measurements

The technological maturity attained by general purpose processors and network interface cards makes today's commodity PCs viable and high performing alternatives to specialized hardware for deploying network devices, such as switches, routers, and generic middleboxes. In addition, the flexibility of the software solution seems to be perfectly in line with the emerging trend towards the data-plane programming abstractions brought by recent proposals such as Openflow and the P4 language. However, if programming abstractions provide the way elementary instructions (primitives) are combined together, the development of such processing primitives is left to the network programmer. Although the type of such functions is strongly domain specific, we can safely assume that the counting primitive is easily required in a great deal of practical contexts. This paper presents a counting framework based on probabilistic sketches and LogLog counters for estimating the cardinality of large multi-sets of data. The proposed data structure is designed to be fast and compact for ready use in the on-line chain of processing of network devices running at multi-gigabit speeds. The complete implementation is provided within the probabilistic data structures (pds) library which has been designed, developed, experimentally assessed, and released as open-source for free download. Although the paper specifically presents two possible use-cases, the pds library can be used in rather general scenarios, even outside the networking domain.