Racewalk: Fast Instruction Frequency Analysis and Classification for Shellcode Detection in Network Flow

Memory corruption attacks still play a significant role in present cybercrime activities, being one of the keystones for worm, virus propagation and building botnets. Moreover,recent disclosures of widespread networking equipment vulnerabilities show that the problem is unlikely to fade away in the near future. The subject of this paper is NOP-sled detection - one of the approaches for detecting malicious code in network flow. NOP-sled is a quite common shellcode preamble used in memory corruption attacks to increase the probability of successful target exploitation. We propose a significant modification of the Stride algorithm which has linear computational complexity and runs over 10 times faster than original Stride and a novel approach for NOP-sled detection using IA-32 instruction frequency analysis and SVM-based classification, which gives significantly less false positives then existing algorithms. Evaluation with Metasploit Framework, CLET,ecl-poly and ADMmutate shows that various NOP-sleds provided by existing shellcode generators have instructions frequency peculiarities, which allow to distinguish between sleds and normal network data with high accuracy while reducing the false positives rate and operating close to 1Gbps speed.