A new instruction overlapping technique for anti-disassembly and obfuscation of x86 binaries

The problem of correctly recovering assembly instructions from a binary has received much attention and both malware and license validation code often relies on various anti-disassembly techniques in order to complicate analysis. One well-known anti-disassembly technique is to use overlapping code such that the disassembler starts decoding from an incorrect byte, but still recovers valid code. The actual code which is supposed to be executed is instead hidden inside a decoy instruction, and is overlapped with the disassembled code. We propose and investigate a new novel anti-disassembly method that allows for exceptional flexibility in the hidden instructions, while at the same time providing a disassembled main path that is executable. This allows the approach to be very efficient against static linear sweep disassembly, but also to be more difficult to detect using dynamic analysis methods. The idea is to utilize highly redundant instructions, e.g., multibyte no-operation instructions, and embed the hidden code in the configurable portions of those instructions. By carefully selecting wrapping instructions, providing overlaps, the hidden execution path can be crafted with great flexibility. We also provide a detection-algorithm, together with testing results, for testing software such that the hidden execution path can be identified.