Interpreting and Comparing Convolutional Neural Networks: A Quantitative Approach

A convolutional neural network (CNN) is sometimes understood as a black box in the sense that while it can approximate any function, studying its structure will not give us any insights into the nature of the function being approximated. In other terms, the discriminative ability does not reveal much about the latent representation of a network. This research aims to establish a framework for interpreting the CNNs by profiling them in terms of interpretable visual concepts and verifying them by means of Integrated Gradient. The interpretability profiling has been done by evaluating the correspondence between individual hidden neurons and a set of human-understandable visual semantic concepts. An integrated gradient-based class-specific relevance mapping approach is proposed that verifies interpretability profiling. Moreover, it is insightful to examine the correlation between the different input classes in terms of an overlapping set of highly active neurons. The result suggests the existence of a structured set of neurons inclined to a particular class. Finally, network ablation is performed to illustrate the performance of the network based on our approach.