Hotcount: A High-Precision Traffic Statistics for Multi-Tenants

Traffic statistics in large-scale data streams play an important role in the network community, and can be used for congestion control, anomaly detection, heavy hitter detection, etc. However, in the context of multi-tenancy, accomplishing the traffic statistics task of multiple tenants with limited resources (CPU, memory, etc) has become one of the major challenges. To solve the challenges above, this paper proposes a multi-tenant-oriented traffic statistics structure, named Hotcount, which can grantee the dynamic allocation of resources in multi-tenancy scenarios. At the same time, Hotcount is also able to realize multi-tenant traffic statistics task, and further improve the accuracy of the traffic statistics task. Hotcount separates the cold and hot flows based on statistics. It uses the hot/cold part to record the hot flow size with high precision and cold flow size with lower precision. With extensive experiment, we proved that the processing speed of Hotcount is similar to that of the original classic algorithm. Meanwhile, it greatly improves the accuracy of traffic statistics tasks. In the per-flow size statistics task, the accuracy is improved by 6.7 to 49.5 times than the original algorithm. In the heavy hitter detection task, the accuracy is improved by 11.3 times to 2065.6 times than the original algorithm even with memory-size constraints.