Detecting Execution Under a Debugger Based on INT3 Opcode Misinterpretation

Debugger is a specialized software that allows developers to test and debug their applications to find and fix errors, bugs, and other flaws. Debugger runs a target application under controlled conditions that allows to track and change execution flow and values of resources and environment such as registers, variables in memory, etc. However, there is a wide range of software that has to be protected from external modification and investigation through debugging. Examples of such software are anti-cheat and anti-tampering systems, digital rights management solutions, applications for carrying out online exams. Execution such software under a debugger increases the possibility of bypassing or disabling checks that can lead to leakage and disclosure of sensitive data, cheating, etc. The paper introduces an approach to detecting application execution under a debugger based on misinterpretation of INT3 opcode Results of testing the proposed approach under various well-known debuggers on Windows and Linux operating systems have been presented.