TCI: A system for distributed network monitoring, troubleshooting and dataset creation

Network traffic monitoring is a very complex task that requires a combination of multiple tools and teams. Very often, detected events must be validated and confirmed, or ongoing detection needs additional detailed data from full packets. All these activities must be done automatically concerning data privacy. This is why we propose a solution in the form of Traffic Capture Infrastructure (TCI), a single system for network traffic capture, investigation, and dataset creation, even in high-speed provider networks. Our system supports extensive user management features to ensure dataset privacy, system integrity, and unified control over many network probes. This paper presents the architecture, main functions, recommendations, and lessons learnt from full packet monitoring in today’s networks. Lastly, we prove the value of this system with several publications that have used our system to create their underlying dataset and network traffic investigation.