DRLFCfuzzer: fuzzing with Deep-Reinforcement-Learning under Format Constraints

With the development of information technology, program vulnerabilities have become more complex and diversified, posing a great threat to the security of computer systems. Among many vulnerability detection methods, fuzzing is an popular method of automatic vulnerability mining which efficiency depends on the quality of generated samples. Some work has introduced reinforcement learning into fuzzing to provide an intelligent scheme for vulnerability mining, but there is still the problem of exploring invalid sample space. In order to solve this problem, this paper proposes a new fuzzer called DRLFCfuzzer, which is based on data boundary segmentation technology and guided by multi-dimensional deep reinforcement learning. We model the fuzzing process as a Markov decision process, add the segmentation of data boundaries and the selection of data blocks on the basis of multi-dimensional mutation, and improve the efficiency of the fuzzing by pruning unnecessary exploration of sample spaces. The experimental results show that DRLFCfuzzer achieves code coverage of 128% to 800% of AFL and 112% to 160% of general deep reinforcement learning fuzzer and more crashes in some programs of the Fuzzer-Test-Suite dataset and three real world programs.