Probability Fusion Decision Framework of Multiple Deep Neural Networks for Fine-Grained Visual Classification

Fine-grained visual classification tasks often suffer from that the subordinate categories within a basic-level category have low inter-class discrepancy and high intra-class variances, which is still challenging research for traditional deep neural networks (DNNs). However, different models extract local parts' features in isolation and neglect the inherent correlations and distribution in high-dimensional space, which limit the single model to achieve better accuracy. In this paper, we propose a novel probability fusion decision framework (named as PFDM-Net) for fine-grained visual classification. More specifically, it first employs data-augmented tricks to enlarge the dataset and pretrain the basic VGG19 and ResNet networks on high-quality images datasets to learn common and domain knowledge simultaneously while fine-tuning with professional skill. Next, refined multiple DNNs with transfer learning are applied to design a multi-stream feature extractor, which utilizes the mixture-granularity information to exploit high-dimensionality features for distinguishing interclass discrepancy and tolerating intra-class variances. Finally, a probability fusion module equipped with gating network and probability fusion layer is developed to fuse different components model with Gaussian distribution as a unified probability representation for the ultimate fine-grained recognition. The input of this module is the various features of multi-models and the output is the fused classification probability. The end-to-end implementation of our framework contain an inner loop about the EM algorithm within an outer loop with the gradient back-propagation optimization of the whole network. Experimental results demonstrate the outperforming performance of PFDM-Net with higher classification accuracy on different fine-grained datasets compared with the state-of-the-arts methods. More discussions are provided to indicate the potential applications in combination with other work.