Performance comparison of four anomaly detectors in detecting self-propagating malware on endpoints

Malware detection has emerged been an active area of research over the last few years. Numerous malware detection techniques have been proposed to combat this rapidly evolving threat. Notable of these detection techniques are rate limiting [10], [11] , the sample entropy based malware detection [8], maximum entropy estimation [9] and the TRW algorithm that employs sequential hypothesis testing [4]. Most of these techniques (except rate limiting) have been designed and tested on the network periphery (e.g., gateway router etc.) Recently, network endpoint comprising home and office computers have become the most prevalent and effective launch pads and carriers of malware infections. Moreover, endpoints represent the last (and sometimes the only effective) line of defense against the spread and detection of malware. Therefore, it is important that contemporary anomaly detectors' performances be evaluated on endpoints and under high and low-rate worm propagation attacks. This paper compares the ab2ove four anomaly detection techniques using real endpoint and worm traffic data collected on operational endpoints.