DSFuzz: deep state of stateful protocol fuzzing

With the rapid development of cloud services and the Internet of Things, more and more services and devices are connected to the Internet. Network security is becoming increasingly important. However, the existing vulnerability detection technology, protocol fuzzing, faces the difficulty of deep-state fuzzing. In this paper, we propose DSFuzz, a generated-based deep state of stateful protocol grey-box fuzzer. There are two main modules in DSFuzz. First, the message sequence dynamic generation module. DSFuzz believes that different execution states should be considered in different network communication states. DSFuzz dynamically traverses the state machine under overlay feedback to generate different message sequences. Message sequences covered by different branches in the same state are considered to be different execution states. The second is the message sequence selection module. Considering the rare coverage branches and the difficult coverage branches, DSFuzz designs the expected feedback calculation formula of the message sequence and selects the message sequence through the Dobby slot machine reinforcement learning algorithm. In this way, DSFuzz can trigger more branch coverage and achieve an acceptable fuzzing efficiency. The experimental results show that compared with the most advanced fuzzers BooFuzz and AFLNET, the branch coverage of different message sequences considering the status of the execution status generation protocol is improved by 22.8% and 89.4%, respectively. This paper also evaluates the effectiveness of the message sequence selection algorithm.