DeepCov: Coverage Guided Deep Learning Framework Fuzzing

The large-scale use of deep learning (DL) technology makes it important to ensure software security in DL systems. All DL systems depend on the DL framework to achieve development tasks. If any vulnerability exists in the DL framework, DL systems will contain potential security risks. In addition, existing DL framework testing methods typically use inconsistencies as the feedback and cannot effectively cover the entire layer, leading to inefficiencies in finding vulnerabilities.In this work, we propose DeepCov, a layer edge coverage based DL framework fuzzing system. DeepCov uses the mutation-based method to generate models that fully explore layer APIs of the DL framework. DeepCov detects three types of bugs, including inconsistency, NaN and crash bugs. Compared to the previous inconsistency guided tool LEMON, DeepCov considers the layer edge coverage metric and uses the multi-dimensional evaluation based seed selection and strategic layer addition operator. By analysing and utilizing the information from models, DeepCov generates mutated models with more API calls of layer types which eventually explore larger model layer space in the DL framework testing. To show the effectiveness of our approach, we evaluated DeepCov on five DL frameworks, i.e., TensorFlow, Pytorch, CNTK, Theano and MXNet. The results show that DeepCov achieves 5 times improvement in the number of tested layer types compared with LEMON. Moreover, 6 crash bugs, 11 unique inconsistency bugs, and a NaN bug were found.