An FPGA-based approach for packet deduplication in 100 gigabit-per-second networks

Network traffic monitoring usually faces the problem of packet duplication, which arises when port mirroring is being used. That is, when traffic is copied from the ports of a switch or a router that are being monitored, to a mirror port where a monitoring probe is attached. Thus, a packet can be copied twice, both at the ingress and egress ports, therefore generating duplicates. Information redundancy caused by packet duplication not only leads to increased workloads at the monitoring probes, but also calls for more disk space to store the network traces. Actually, packet duplication may increase 100% the monitoring load. There are different sorts of packet duplication; in this paper we focus on switching duplication, because it is the most common in a network monitoring scenario, where the network probe is attached to a core switch. We present a high performance FPGA architecture that is able to detect and remove duplicated packets in 100 Gbit/s networks. It is based on a 64-bit key and a BRAM-based shift register that allows us to build an element-based sliding window of size up to 79,872 elements. The design targets the Xilinx Virtex UltraScale family, using the integrated 100G Ethernet Subsystem available in such devices, and it has been tested on a VCU108 evaluation kit.