Modeling of Checking Program Flow Control Transitions when Executing a Return Instruction

Buffer overflow vulnerabilities have been known for a long time, are widespread and relevant today. To protect against buffer overflows, developers of operating systems usually use standard mechanisms of operating systems, but they are not very effective in attacks using return-oriented programming techniques. The article proposes a model for checking the legitimacy of program control flow transitions when executing a return instruction, based on an access graph. The article also provides a model for checking the legitimacy of program control flow transitions when executing a return instruction, based on the use of a shadow stack, a model for generating a program control flow. Modeling was carried out in the specification language based on set theory, first-order logic and temporal logic of actions, models were checked using the Model Checking method. The constructed models can be refined to the required level of abstraction, used to create mechanisms for protecting operating systems based on Linux from attacks using the return-oriented programming technique.