Evaluation of T-Way Testing of DNNs in Autonomous Driving Systems

A Deep Neural Network (DNN) model is used to perform intelligent, safety-critical tasks in Autonomous Driving Systems (ADS). In our prior work, we proposed a combinatorial testing approach to test DNN models used to predict a car's steering angle. We generate test images by applying a set of combinations of basic image transformations. In this paper, we report a preliminary study that compares the performance of synthetic images generated using a combinatorial approach to DeepTest, a state-of-the-art tool that aims at generating test inputs that maximize neuron coverage. We present an experimental evaluation by measuring and comparing the neuron coverage achieved using the two approaches. Two pre-trained DNN models from the Udacity driving challenge are used as the subject DNNs. The results suggest that the combinatorial approach performs better than the DeepTest approach in generating valid synthetic images and covering an additional number of neurons.