PrIDE: A Protocol-Independent De-Duplication Engine for Packet Recording

Packet recording or capturing is one of the most useful tools for network forensics and surveillance. Since a storage system is of a limited size, de-duplication can be used to save disk space. In this article, we present a new scalable de-duplication engine for packet recording that can eliminate redundant contents over multiple packets. Unlike previous work, our proposed scheme is designed for packet-level de-duplication to support any kinds of network from the current Internet to emerging networks. We also present a new fast chunking method and a new indexing scheme that enable multiple engine instances to execute in parallel. We implement the de-duplication engine, and experimental results show that our proposed scheme can remove up to 65 percent of the packet contents in a real campus network. We also confirm that its throughput scalably increases with the number of CPU cores, which means that the proposed scheme can be implemented in a wide range of computing devices from small home gateways to high-end servers.