Spector: Automatically Analyzing Shell Code

Detecting the presence of buffer overflow attacks in network messages has been a major focus. Only knowing whether a message contains an attack, however, is not always enough to mitigate the threat. It may also be critical to know what it does. Unfortunately, shell code is written in low-level assembly language, and can be obfuscated. The current method of analyzing shell code, manual reverse engineering, is time-consuming, requires significant expertise, and would be nearly impossible for a wide-scale polymorphic attack. In this paper, we introduce Spector, a symbolic execution engine that extracts meaningful high-level actions from shell code. Spector's high-level output helps facilitate attack mitigation and classification of different payloads that have the same behavior. To evaluate Spector, we tested it with over 23,000 unique payloads. It identified eleven different classes of shell code, and processed all the payloads in just over three hours. Spector also successfully classified polymorphic instances of the same shell code.