IDROP: Intelligently detecting Return-Oriented Programming using real-time execution flow and LSTM

Return-Oriented Programming (ROP) has become one of the most widely used attack techniques for software vulnerability exploitation. Existing ROP detection methods fall into two types: hardware-based methods and software-based methods. The former is strongly dependent on specific hardware architectures and difficult to deploy. Although the latter can alleviate these problems, limited by the selection of features and thresholds, it cannot effectively discover neither variant ROP nor delayed ROP. In this work, we propose an intelligent detection method at runtime and implement the corresponding prototype system, IDROP, which uses real-time execution flow and LSTM to discovery ROP and its variants. Specifically, IDROP analyzes the differences between program execution flows that are independent of the ROP feature thresholds. Firstly, the Aspect Oriented Programming (AOP) is utilized to instrument the tested program, and the sliding window mechanism is applied to screen out suspicious program execution flow snapshots. Then, these suspicious execution flow snapshots are vectorized through data representation techniques. Finally, we build and train an LSTM model to discover ROP. Furthermore, we evaluate the performance of IDROP on a dataset consisting of 6000+ samples. The experimental results show that IDROP is effective in detecting ROP attacks, variant ROP and delayed ROP with an accuracy of 98%, 93% and 80%, respectively. In addition, IDROP has negligible space overhead and low performance overhead, which is similar to that of only using Pin for detection (about additional 2.5 times the program execution time before instrumentation).