Grafting Heterogeneous Neural Networks for a Hierarchical Object Classification

Convolutional neural networks (CNNs) are deep learning architectures used for image classification that have been improved in recent years to increase their accuracies and reduce their computation times. Hierarchical approaches are based on a step-by-step strategy and aim to optimize performance on difficult tasks by solving successive subtasks. The gain provided by these solutions must be relativized with the explosion in the number of parameters they imply, which makes their implementation on embedded systems difficult. New constraints also appear in the choice of the architectures of the branches when one seeks to have a global network providing predictions at different levels. We propose a strategy that allows the merging of heterogeneous CNNs by following a hierarchical approach in which the information extracted by first-level networks can be fed back at any location into second-level networks. Despite the differences in the number and size of the feature maps, such grafting can be done by using clustering, dimension reduction, and interpolation techniques. This strategy eliminates the computational redundancy induced by the recalculation of low-level features. The proposed grafting approach significantly reduces the inference time of the second network without impacting the accuracy. Tests performed on MNIST, CIFAR-10, and PlantVillage datasets with several CNNs illustrate the possibility of implementation in various situations. Our solution allows us to consider in an innovative way the implementation of hierarchical solutions on devices with limited capacities.