Common Program Similarity Metric Method for Anti-Obfuscation

Program similarity metrics, especially malware similarity metrics, have long been an area of active research. However, code obfuscation techniques bring many challenges to similarity analysis. Most prior techniques lack extensibility since they focus on problems of only one particular type, such as malicious programs on a specific platform. Moreover, some of these techniques cannot measure the similarity between the various components of two programs, and some cannot accommodate code obfuscation techniques, particularly for control flow obfuscation. To address these limitations, we present the new concept of the reductive instruction dependence graph (RIDG), which is platform-independent and stable throughout most code obfuscation processes. In addition, we propose a four-level similarity schema that is based on RIDG for measuring the similarity between two programs, which can be used to find the similarity between the components of programs. We use the proposed program similarity metric method to measure similarities among 100 different programs, and we evaluate the anti-obfuscation ability of this method by using four popular code obfuscation techniques. The experimental results show that the proposed method can overcome the limitations described above.