Grey-box Fuzzing With Deep Reinforcement Learning And Process Trace Back

Grey-box fuzzing, as a software testing technique, can find possible program bugs such as memory leaks, assertion failures and invalid input by generate random data then input it into a program, and monitor program exceptions. With program running, grey-box fuzzing collected the branch information in order to guide choosing next seeds. In this paper, we try to use the concept of Markov decision processes to formalize grey-box fuzzing as a deep reinforcement learning problem, and use process trace back(Intel Process Trace) to collect branch information in order to improve its efficiency toward binary programs. The experiments show this approach can outperform baseline random fuzzing and gain performance improvement.