Feature-Based Interpretation of Image Classification With the Use of Convolutional Neural Networks

Deep neural networks have made remarkable contributions in many fields such as computer vision and natural language processing. However, due to their complex internal structures, for a long time, there is a lack of trust in deep neural networks, particularly in the domains like healthcare, finance, traffic safety, etc. To explain the internal mechanism of deep neural networks, a series of interpretable models, which mainly focus on enhancing the interpretability of deep learning models, has been proposed. This study pursues to data interpretability by analyzing the significance of image features with the goal of mining essential information contained in various visual features. Super-pixel segmentation is firstly applied to divide the image into several coherent visual feature regions, and these regions are subsequently overwritten by certain marks. Next, image occlusion is realized by modifying the pixel values of these feature regions according to the marks. Finally, we conduct comparative analytics of the classification results between the original data and occluded data to scrutinize the importance of key visual features. A collection of experimental studies is conducted and the results consistently demonstrate that the proposed method enables to observe the classifier’s dependence on different features, offering a new perspective on interpretability rooted in data analytics.