Contents Model optimization and statistical inference have played a central role in various applications of computational intelligence, data analytics, and computer vision. Traditional approaches are usually based on model-centric learning. That is, even after model training, it is still required to design proper algorithms and to specify hand-crafted parameters for optimization and inference. Recently, discriminative learning has demonstrated its power for process-centric learning. Taking domain expertise and problem structure into account, problem-specific deep architectures can be formed by unfolding the model inference as an iterative process, and the parameters of the optimization process can then be learned from training data. These solutions are closely related with bilevel optimization, partial differential equation (PDE), as well as meta learning, and can provide new insights into the studies of versatile statistical and optimization models, such as sparse representation, structured regression, and conditional random fields. Moreover, generic deep network architectures are often referred to as “black-box” methods, while discriminative process-centric learning can provide a new perspective for the understanding and development of generic deep architectures. To sum up, connecting discriminative learning with model optimization and inference is not only helpful in analyzing convergence and generalization of deep architectures but also offers new perspectives for understanding and developing generic deep learning models.
This special issue aims at consolidating the recent progress in theoretical and methodological developments and real-world applications to facilitate this direction. We received a large number of submissions that represent a broad spectrum of research in discriminative learning and model-driven network architecture design. Among them, fifteen articles were accepted after a rigorous but rewarding review process coordinated by the guest editors. The accepted articles have been organized into three groups.
The first group is about discriminative sparse and low-rank optimization models, which have been extended to handle data with multiple forms such as a mixture of Gaussians, multimanifold, low-rank matrix, and tensor.
Fisher’s Discriminant on Mixture of Gaussians: Zheng et al. extended Fisher’s discriminant analysis to a mixture of Gaussians, where a mixture of absolute generalized Rayleigh quotients is derived with Bayes error upper bound estimation on a mixture of Gaussians. Furthermore,
L_{1} L_{1} L_{1} Double Nonconvex Nonsmooth Rank Minimization: For the efficient recovery of the low-rank matrix, Zhang et al. suggested a double nonconvex nonsmooth rank (DNNR) minimization model, where a flexible weighted NNR relaxation is introduced on the nonconvex singular value function (SVF). An iteratively reweighted SVF (IRSVF) optimization algorithm is presented to solve DNNR, which is proved to be monotonically nonincreasing. With some milder assumptions, the global convergence of IRSVF can also be guaranteed with the Kurdyka–Łojasiewica (KŁ) property. Experiments on matrix completion show that DNNR performs favorably against the competing methods.
Supervised Discriminative Sparse Principal Component Analysis (SDSPCA): Feng et al. presented an SDSPCA model, where group sparsity is enforced to constrain the orthogonal projection matrix. Then, a reconstruction term of the class indicator matrix is also deployed to exploit the supervision information. An alternating optimization is developed to solve SDSPCA, and it is proved that the model objective is nonincreasing along with the iterations. The results on gene selection and tumor classification indicate its effectiveness.
Tensor Probabilistic Linear Discriminant Analysis (TPLDA): Ju et al. extended probabilistic linear discriminant analysis for handling tensor data, namely, TPLDA, where each tensor can be decomposed into three parts, i.e., shared subspace, individual subspace, and noise. While the first two parts are separately depicted by probabilistic subspaces, the noise part adopts the form of multivariate Gaussian. All the model parameters are then learned using the Bayesian inference and tensor Cande–Comp/PARAFAC (PARAllel FACtors) decomposition. Experiments show that TPLDA is promising in separating individual subspace components.
Semisupervised Discriminant Analysis on MultiManifold: Xu et al. investigated the model of semisupervised discriminant analysis on multimanifold. It models human actions with multimanifold subspaces with each for a specific action category. Then, a semisupervised discriminant multimanifold analysis can be formulated as an unconstrained optimization program and solved using the spectral projected gradient (SPG) method. The results show that the proposed method performs favorably against the competing methods.
In the second group, four articles are included to connect deep networks with hypergraph, prior driven models, and metric learning, respectively.
Hypergraph-Induced Convolutional Network: Shi et al. developed a hypergraph-induced convolutional network (CNN) by utilizing high-order correlation in visual data. In order to apply hypergraph-induced CNN to visual classification, a hypergraph is first constructed for modeling the relationship in visual data, and convolution operator and optimization are then modified to take high-order relationship into account. The results show that hypergraph-induced CNN is effective in visual classification.
Prior Driven Network Design: For haze removal, Liu et al. aggregated both a prior driven model and a data-driven network to design a residual architecture for transmission propagation and scene radiance estimation. An energy-based investigation is further given to analyze the propagation behavior of the aggregated network. Furthermore, by incorporating with task-aware image separation, the model can be extended to address other image enhancement tasks such as underwater image enhancement and single-image rain removal.
Joint Distance Metric Learning and Deep Representation Learning: For person reidentification, Yang et al. incorporated distance metric learning and deep representation learning with an end-to-end trainable framework. In particular, a structural metric learning objective is presented for mining both hard positive and hard negative samples, and a global loss term is further included to enhance the generalization ability of the learned network. The results show that the proposed approach performs favorably against the state-of-the-art methods.
Combining Metric Learning With Deep Network: Liu et al. also studied person reidentification by combining discriminative metric learning with deep networks. They presented a framework for joint learning of local distance metric and deep feature representation, where the local metric is introduced to enhance the discriminative and generalization ability. Experiments on the benchmark data sets demonstrate its effectiveness.
The third group contains six articles that exploit discriminative learning and model optimization to real-world applications such as gaze estimation, visual tracking, video captioning, imbalanced clustering, and positive-unlabeled classification.
Multiview Multitask Gaze Estimation: Lian et al. estimated gaze point as well as gaze direction by using multiview cameras. For multitask learning, they presented a partially shared convolutional neural network architecture. To facilitate the studies on multiview gaze estimation, a multiview gaze tracking data set is also built, which is noted to have the largest numbers of both subjects and images. Extensive experiments are conducted to analyze the effect of factors such as network architecture and view number.
Structural Support Vector Machine for Visual Tracking: Zheng et al. improved structural support vector machine (SSVM) for visual tracking from two aspects, resulting in a spatially regularized SSVM (SRSSVM). First, a spatial regularization prior is employed to restrict the learned classifier to have the same size as the target. Second, explicit feature map is adopted to approximate the intersection kernel. The resulting SRSSVM model can then be efficiently solved the dual-coordination descent algorithm. The results on Online Tracking Benchmark-50 (OTB50), OTB100, and Visual Object Tracking Challenge 2016 (VOT2016) validate its effectiveness and robustness in visual tracking.
Unified Dynamic Collaborative Tracking: To address the model drift issue in correlation with filter-based trackers, Zhu et al. presented a unified dynamic collaborative tracking framework for flexible and robust position prediction. In the proposed model, distracter suppression and maximum margin relation terms are introduced to suppress distracting regions and enhance discriminative ability. An online CUR filter is further incorporated for alleviating model drift. Experiments indicate that the proposed model achieves the state-of-the-art performance on OTB and VOT benchmarks.
Encoder–Decoder Network for Video Captioning: For video captioning, the deterministic hidden states in conventional decoders are insufficient in characterizing the uncertainties of video content. Therefore, Song et al. improved the encoder–decoder video captioning network by presenting multimodal stochastic recurrent neural networks (MS-RNNs), multimodal long short-term memory (M-LSTM), and backward stochastic (S-LSTM). In particular, MS-RNNs are adopted for modeling the uncertainty with latent stochastic variables, while M-LSTM and backward S-LSTM are deployed to, respectively, capture high-level representation and support uncertainty propagation. Experiments on two data sets demonstrate the effectiveness of the proposed method.
Global Balanced Clustering: For balanced clustering, the size of each cluster is constrained to be nearly equal. Han et al. proposed a global balanced clustering (GBC) model, where global discriminative partitioning and distribution entropy are combined for simultaneously capturing the local and global characteristics of data. They further presented two clustering models, i.e., local and global balanced spectral clustering (LGB-SC) and local and global balanced learning-based clustering (LGB-LL). Experiments show that LGB-LL and LGB-SC can make a good tradeoff between clustering quality and balanced clusters.
Robust Positive-Unlabeled Classification: Ren et al. developed a robust learning framework to unify area under the curve (AUC) maximization, outlier detection, and feature selection. Generalization error bounds are also given to provide some insights into theoretical performance and practical guidance of the method. Empirical results show that the proposed method performs well in surgical site infection (SSI) and EEG seizure detection.
Discriminative learning is a long-standing topic in machine learning but is a newly emerging research trend for model optimization and statistical inference. Our hope is to encourage the continuous development and extensive applications of this field. We would like to thank all authors for their contributed works. We are also indebted to the Editor-in-Chief for his consistent support and valuable guidance throughout the application, review, and publication process. Finally, we thank all reviewers for their constructive comments, which are enormously valuable not only for us in selecting high-quality accepted articles but also for the authors in improving their works.