Are Coverage Criteria Meaningful Metrics for DNNs?

The wide deployment of Deep Neural Networks (DNNs), though achieving great success in many domains, has severe safety concerns. Inspired by testing criteria from traditional software engineering, various coverage criteria have been proposed to ensure the safety of DNNs. However, the validity of coverage criteria was questioned in related researches. In this paper, we evaluate the performance of dominating coverage criteria in two aspects: 1) distinguishing different qualities of test sets, 2) improving the safety and robustness of DNNs. The evaluation result confirms that coverage criteria are meaningful metrics for DNNs. Specifically, the way for improving robustness is contrary to the previous assumption: the higher coverage criterion score, the better. In addition, we propose a new coverage criterion called Independence Neuron Coverage (INC) which is finer grained to capture DNNs' subtle behaviour. Experiments show that INC is efficient and performs better than other evaluated coverage criteria in both aspects.