A Data Flow-Aware Control Flow Flattening Deobfuscation Approach

Control Flow Flattening (CFF), a prevalent obfuscation method employed in malicious software, transforms program structure to circumvent detection mechanisms. However, state-of-the-art static-analysis-based CFF deobfuscators have a low success rate when dealing with compilation-optimized binaries, as they rely on flow-insensitive static analysis. To analyze malware more efficiently, this paper first systematically examines the limitations of existing static-analysis-based CFF deobfuscators. Based on these observations, we propose and implement a data flow-aware CFF deobfuscator, named FlowSight, to effectively deobfuscate compilation-optimized binaries. Specifically, FlowSight identifies OO Blocks through forward data flow analysis and recovers the OO Block control flow graph (CFG) using backward data flow analysis. To demonstrate the effectiveness of FlowSight, we conduct experiments showing that the proposed system outperforms state-of-the-art static-analysis-based deobfuscation tools, with the success rate increasing from 39.77% to 79.04%.