RADAR: An automated system for near real-time detection and diversion of malicious network traffic

This paper presents the automated RADAR system for the malicious traffic detection and the near realtime traffic diversion. RADAR is designed to detect and divert suspicious/malicious traffic (within an enterprise network) to safe locations for further analysis. Current practice requires the manual redirection of malicious traffic, incurs a significant amount of human effort, and results in major delay in response. The RADAR system includes network monitoring and traffic filtering/diversion with threat detection, ruleset generation, rule dissemination (similar to Border Gateway Protocol (BGP) Flowspec) and threat analysis capabilities. RADAR decouples Flowspec features from BGP routing and provides the flexibility to use any standard routing protocol such as BGP and OSPF to divert the traffic. BGP is used for the purpose of demonstration. A scalable and hybrid network testbed is developed to emulate an enterprise network, where normal and malicious traffic flows, i.e., denial of service (DoS) attacks, are generated. A NetFlow based network monitoring tool is used to provide near real-time traffic specification feedback and a flow based intrusion detection system is used to detect DoS TCP SYN attacks. Ruleset generation and diversion mechanisms are developed to construct BGP updates for the redirection of malicious flows. The successful diversion of malicious traffic is demonstrated with Cisco and Linux routers for testing different scenarios in the network testbed.