Squashing Resource Exhaustion Bugs with Black-Box Fuzzing and Reinforcement Learning

For a software system to be reliable, it must manage its resources properly. Failure to do so will result in unreliable behaviour: an application that leaks memory will eventually crash, a packet source that overloads a queue may cause other systems to fail, a process that consumes too many CPU cycles will degrade the performance of other processes and so on. Resource leaks or resource exhaustion are difficult to discover during testing as it may happen slowly over a long time. One approach for discovering issues with reliability, security and robustness is fuzzing (short for fuzz testing). Fuzzing can take many forms, depending on what type of system is to be tested and what kinds of bugs one is after. Black-box fuzzing is arguably the most flexible approach to fuzzing. Unfortunately, it suffers from a low efficiency that makes it slow at finding bugs such as resource leaks. In this paper we explore the topic of black-box fuzzing by modeling it as a multi-armed bandit problem, an important subclass of the general reinforcement learning problem. We believe that by utilizing a reinforcement learning framework, black-box fuzzing can be better understood and attention can be drawn to the field, which deserves to be studied more. We also implement a fuzzer according to our model and evaluate it against a toy implementation of a simple protocol with a known resource leak. Lastly, we apply our fuzzer in a real-world case study against two widely distributed implementations of the Link Layer Discovery Protocol (LLDP), a key component in critical infrastructure applications such as network management and network automation. Our results show that our fuzzer gradually learns how to effectively trigger the resource leak in the toy implementation, thereby speeding up the bug discovery process. In the case study, the fuzzer struggles to learn from the observations it makes about the test target. We believe this to be because of excessive delays between the actions the fuzzer takes during testing and their corresponding effects. Despite this, our fuzzer still manages to find one resource leak in each of the two LLDP implementations, one of which was previously unknown. With this paper, we have taken the first steps towards a better understanding of black-box fuzzing and that a new generation of smart and highly efficient black-box fuzzers is within reach.