Layer-Wise Analysis of Neuron Activation Values for Performance Verification of Artificial Neural Network Classifiers

Object classification in dynamic, uncontrolled environments is one of the functional elements of safety-critical Autonomous Systems. It is crucial to develop methods for the specification and verification of these elements, and the associated algorithms, in order to gain confidence in the overall safety of Autonomous Systems and their functional and behavioural correctness and adequacy. Artificial Neural Network (ANN) object classifiers must therefore be assured and need to be verified with respect to requirements. A classifier might be required to generalize to a satisfactory extent, in the sense that its classification performance must be maintained at an acceptable level when the input data differs from the training data. This requirement would apply when data received during operation is drawn from a different distribution to the training data. The specification and verification of classifier generalization capability can be based on measures of the dissimilarity between operational and training data. A requirement could state the permitted forms of the relationship between classification performance and a data dissimilarity measure. We have previously proposed such a dissimilarity measure, which we have termed the Neuron Region Distance (NRD). The NRD is a function of network activation values. In this paper, we analyze neuron activation values layer-by-layer across a neural network. This is in order to advance our progress towards the conception of a novel, generalized form of the NRD. This new measure is called the Per Neuron Ranking (PNR) measure. The activation value analysis provides insight into the required formulation of the PNR measure.