Probabilistic Class-Specific Discriminant Analysis

In this paper we formulate a probabilistic model for class-specific discriminant subspace learning. The proposed model can naturally incorporate the multi-modal structure of the negative class, which is neglected by existing methods. Moreover, it can be directly used to define a probabilistic classification rule in the discriminant subspace. We show that existing class-specific discriminant analysis methods are special cases of the proposed probabilistic model and, by casting them as probabilistic models, they can be extended to class-specific classifiers. We illustrate the performance of the proposed model, in comparison with that of related methods, in both verification and classification problems.


I. INTRODUCTION
Class-Specific Discriminant Analysis (CSDA) [1], [2], [3], [4], [5], [6], determines an optimal subspace suitable for verification problems, where the objective is the discrimination of the class of interest from the rest of the world. As an example, let us consider the person identification problem, either through face verification [1], or through exploiting movement information [7]. Different from person recognition, which is a multi-class classification problem assigning a sample (facial image or movement sequence) to a class in a pre-defined set of classes (person IDs in this case), person identification discriminates the person of interest from all rest people.
While multi-class discriminant analysis models, like Linear Discriminant Analysis (LDA) and its variants [8], [9], [10], can be applied in such problems, they are inherently limited by the adopted class discrimination definition. That is, the maximal dimensionality of the resulting subspace is restricted by the number of classes, due to the rank of the between-class scatter matrix. In verification problems involving two classes LDA and its variants lead to one-dimensional subspaces. On the other hand, CSDA by expressing class discrimination using the out-of-class and intra-class scatter matrices is able to define subspaces the dimensionality of which is restricted by the the number of samples forming the smallest class (which is usually the class of interest) or the number of original space dimensions. By defining multiple discriminant dimensions, CSDA has been shown to achieve better class discrimination and better performance compared to LDA [2], [4], [5].
While the definition of class discrimination in CSDA and its variants based on the intra-class and out-of-class scatter matrices overcomes the limitations of LDA related to the dimensionality of the discriminant subspace, it overlooks the structure of the negative class. Since in practice samples forming the negative class belong to many classes, different from the positive one, it is expected that they will form subclasses, as illustrated in the example of Figure 1. Class discrimination as defined by CSDA and its variants disregards this structure.
In this paper, we formulate the class specific discriminant analysis optimization problem based on a probabilistic model that incorporates the above-described structure of the negative class. We show that the optimization criterion used by standard CSDA and its variants corresponts to a special case of the proposed probabilistic model, while new discriminant subspaces can be obtained by allowing samples of the negative class to form subclasses automatically determined by applying (unsupervised) clustering techniques on the negative class data. Moreover, the use of the proposed probabilistic model for class-specific discriminant learning naturally leads to a classification rule in the discriminant subspace, something that is not possible when the standard CSDA criterion is considered.

II. RELATED WORK
Let us denote by S p = {x 1 , . . . , x Np } a set of N p Ddimensional vectors representing samples of the positive class and by S n = {x Np+1 , . . . , x N }, where N = N p + N p , a set of N n vectors representing samples of the negative class. In the following we consider the linear class-specific subspace learning case and we will describe how to perform nonlinear (kernel-based) class-specific subspace learning following the same processing steps in Section III-C. We would like determine a linear projection W ∈ R D×d , mapping x i to arXiv:1812.05980v2 [cs.
LG] 24 Jan 2019 a d-dimensional subspace, i.e. z i = W T x i that enhances discrimination of the two classes.
Class-specific Discriminant Analysis [2] defines the projection matrix W as the one maximizing the following criterion: where D n (W) and D p (W) are the out-of-class and intra-class distances defined as follows: and m is the mean vector of the positive class, i.e. m = 1 Np xi∈Sp x i . That is, it is assumed that the positive class is unimodal and W is defined as the matrix maping the positive class vectors as close as possible to the positive class mean vector, while it maps the negative class vectors as far as possible from it, in the reduced dimensionality space R d .
The criterion J (W) in (1) can be expressed as: where T r(·) is the trace operator. S n ∈ R D×D and S p ∈ R D×D are the out-of-class and intra-class scatter matrices defined in the space R D : and W is obtained by solving the generalized eigen-analysis problem of S n w = λS p w and keeping the eigenvectors corresponding to the d largest eigenvalues [11]. Here, we should note that, since the rank of S p is at most N p − 1 (and considering that usually N n > N p ), the dimensionality of the learnt subspace d is restricted by the number of samples forming the positive class (or by the dimensionality of the original space D), i.e. d ≤ min(N p − 1, D). In the case where S n is singular, a regularized version of the above problem is solved by usingS n = S n + I, where I ∈ R D×D is the identity matrix and is a small positive value. A Spectral Regression [12] based solution of (4) has been proposed in [4], [5]. Let us denote by w an eigenvector of the generalized eigen-analysis problem S n w = λS p w with eigenvalue λ. By setting Xw = v (X being the data matrix), the original eigen-analysis problem can be transformed to the following eigen-analysis problem P n v = λP p v, where P n = e n e T n − 1 Np e n e T p − 1 Np e p e T n + 1 Here e p ∈ R N and e n ∈ R N are the positive and negative class indicator vectors. That is, e p,i = 1 if x i ∈ S p and e p,i = 0 for x i ∈ S n , and e n,i = 1 − e p,i . Based on the above, the projection matrix W optimizing (4) is obtained by applying a two-step process: • Solution of the eigen-analysis problem P n v = λP p v, leading to the determination of a matrix V = where v i is the eigenvector corresponding to the i-th largest eigenvalue. • Calculation of W = [w 1 , . . . , w d ], where: It has been shown in [13] that by exploiting the structure of P p and P n , their generalized eigenvectors v can be directly obtained using the labeling information of the training vectors. However, in that case the order of the eigenvectors is not related to their discrimination ability. Finally, it has been shown in [14] that the class-specific discriminant analysis problem in (1) can be casted as a low-rank regression problem in which the target vectors are the same as those defined in [13], providing a new proof of the equivalence between class-specific discriminant analysis and class specific spectral regression.
After determining the data projection matrix W, the training vectors x i , i = 1, . . . , N are mapped to the discriminant subspace z i = W T x i . When a classification problem is considered, a classifier is trained using z i . For example, a linear SVM is used in [5] that is trained on vectors d i = |z i − µ|, where µ is the mean vector of the positive samples in the discriminant subspace and the absolute value operator is applied element-wise on the resulting vector. Due to the need of training an additional classifier, class-specific discriminant analysis models are usually employed in class-specific ranking settings, where test vectors x * j are mapped to the discriminant subspace z * j = W T x * j and are subsequently ordered based on their distance w.r.t. the positive class mean d j = z * j − µ 2 .

III. PROBABILISTIC CLASS-SPECIFIC LEARNING
In this section, we follow a probabilistic approach for defining a class-specific discrimination criterion that is able to encode subclass information of the negative class. We call the proposed method Probabilistic Class-Specific Discriminant Analysis (PCSDA). PCDA assumes that exists a data generation process for the positive class following the distribution: where m is the mean vector of the positive class and Φ p is the covariance of the underlying data generation process. A (multi-modal) negative class is formed by subclasses the representations of which are drawn from the following distribution: Here y denotes the representations of the negative subclasses and Φ n is the covariance of the negative subclass generation process. Samples of the negative subclasses are drawn from the following distribution: where Φ w is the covariance of the underlying data generation process for each subclass of the negative class. As we will show later, a special case of the above definition for the negative class is when each subclass is formed by one sample, leading to the class discrimination criterion used by the standard CSDA and it's variants. Given a set of positive samples S p = {x i , . . . , x Np } the probability of correct assignment under our model (under the assumption that the data are i.i.d.) is equal to: Let us assume that negative class is formed by K subclasses, i.e. S n = {S 1 , . . . , S K }, each having a cardinality of M = N n /K. We will show how to relax this assumption in the next subsection. Then the probability of assigning each of the negative samples to the corresponding subclass is equal to: By centering the training samples to the positive class mean (this can always be done by setting and In the above, S p is the scatter matrix of the positive class, w is the within-subclass scatter matrix of the k-th subclass of the negative class, i.e. S (k) n is the scatter matrix of the k-th subclass w.r.t. the positive class, i.e. S (k) n =x kx T k , wherex k is the mean vector of the k-th subclass of the negative class, i.e.x k = 1 M xi∈S k x i . Since the assignment of the negative samples to subclasses is not provided by the labels used during training, we define them by applying a clustering method (e.g. K-Means) on the negative class vectors.

A. Training phase
The parameters of the proposed PCSDA are the covariance matrices Φ p , Φ n and Φ w defining the data generation processes for the positive and negative classes. These parameters are estimated by maximizing the (log) probability of correct assignment of the vectors x i , i = 1, . . . , N : where S n and S w are the total scatter matrices of the negative subclasses, i.e., n . By substituting (16) and (17) in (15) the optimization function takes the form: The saddle points of L with respect to Φ p , Φ n and Φ w lead to: Note that when N n = K (21) takes an indeterminant form (i.e. 0/0). However, in this case each negative subclass is formed by one sample and by definition Φ w = 0. This is discussed more in subsection III-C. Combining (20) and (21) we get: By combining (21) and (22) we can define the total scatter matrix for the negative class: Let us denote by Q ∈ R D×D the eigenvectors of the generalized eigen-analysis problem: The columns of Q diagonalize both Φ p and Φ t , i.e.
Let W ∈ R D×D be the solution of the eigen-analysis problem Np K S n + Np Nn S w w = λS p w. From (19) and (25): where Λ p = W T S p W is a diagonal matrix scaling the columns of W −T to calculate V. Combining (24), (25) and (26): where Λ t = W T S t W. Thus, V and ∆ t can be computed by solving an eigen-analysis problem defined on the input vectors x i , i = 1, . . . , N .
In the above we set the assumption that the number of samples forming the negative classes is equal. This assumption can be relaxed following one of the following approaches. After assigning all negative samples to subclasses and calculating the conditional distributions of each subclass, one can sample M vectors from these distributions. Alternatively, one can calculate the total within-subclass matrix of the negative class S w and define the subclass scatter matrices as S (k) w = 1 K S w . Note that for the calculation of the model's parameters (Eqs. (19) and (23)), only the total scatter matrices S p , S w and S n are used.

B. Test phase
After the estimation of the model's parameters, a new sample represented by the vector x * can be evaluated. The posterior probabilities of the positive and negative classes are given by: The a priori class probabilities P (c p ) and P (c n ) can be calculated by the proportion of the positive and negative samples in the training phase, i.e. P (c p ) = N p /N and P (c n ) = N n /N . Depending on the problem at hand, it may be sometimes preferable to consider equiprobable classes, i.e. P (c p ) = P (c n ) = 1/2, leading to the maximum likelihood classification case. The class-conditional probabilities of x * are given by: and In the case where we want z * ∈ R d , d < D, we keep the dimensions of z * corresponding to the d largest diagonal elements of ∆ t . Combining (28)-(31) the ratio of class posterior probabilities is equal to: and can also be used to define a classification rule: classifying x * to the positive class if g(x * ) > 0 and to the negative class otherwise. In class-specific ranking settings one can follow the process applied when using the standard CSDA approach. First, test vectors x * j are mapped to the discriminant subspace z * j = W T x * j and z * j , i = 1, . . . , N are obtained. Then, z * j 's are ordered based on their distance w.r.t. the positive class mean d j = z * j − µ 2 .

C. CSDA variants under the probabilistic model
A special case of the PCSDA can be obtained by setting the assumption that each negative sample forms a negative subclass, i.e. K = N n and M = 1. In that case Φ w = 0, Φ t = Φ n , the negative samples are drawn from a distribution P (x) ∼ N (·|m, Φ n ), and W is calculated by solving for S n w =λS p w, whereλ = λN n /N p , i.e., we obtain the class discrimination definition of CSDA. The Spectral Regressionbased solutions of CSDA in [4], [5], [13] and the low-rank regression solution of [14] use the same class discrimination criterion and, thus, correspond to the same setting of PCSDA. Since the discrimination criterion used in CSDA is a special case of the proposed probabilistic model, all the abovementioned methods can be extended to perform classification using g(·) in (33).

D. Non-linear PCSDA
In the above analysis we considered the linear class-specific subspace learning case. In order to non-linearly map x i ∈ R D to z i ∈ R d traditional kernel-based learning methods perform a non-linear mapping of the input space R D to the feature space F using a function φ(·), i.e. x i ∈ R D → φ(x i ) ∈ F. Then, linear class-specific projections are defined by using the training data in F. Since the dimensionality of F is arbitrary (virtually infinite), the data representations in F cannot be calculated. Traditional kernel-based learning methods address this issue by exploiting the Representer theorem and the non-linear mapping is implicitly performed using the kernel function encoding dot products in the feature space, i.e.
As has been shown in [16] the effective dimensionality of the kernel space F is at most equal to L = min(D, N ) and, thus, an explicit non-linear mapping This is achieved by using Φ = Σ 1 2 U T , where U and Σ contain the eigenvectors and eigenvavlues of the kernel matrix K ∈ R N ×N [17]. Thus, extension of PCSDA to the non-linear (kernel) case can be readily obtained by applying the above-described linear PCSDA on the vectors φ i , i = 1, . . . , N . For the cases where the size of training set is prohibitive for applying kernel-based discriminant learning, the Nyström-based kernel subspace learning method of [18] or nonlinear data mappings based on randomized features, as proposed in [19], can be used. Here we should note that the application of K-Means using in R L corresponds to the application of kernel K-Means in the original space R D .

IV. EXPERIMENTS
In this Section we provide experimental results illustrating the performance of the proposed PCSDA in comparison with existing CSDA variants. For each dataset, we formed classspecific ranking problems for each class using the class data as positive samples and the data of the remaining classes as negative samples. In all the experiments the data representations are first mapped to the PCA space preserving 98% of the spectrum energy and subsequently non-linearly mapped to the subspace of the kernel space (as discussed in Section III-D). We used the RBF kernel function setting the value of σ equal to the mean distance value between the positive training vectors. The discriminant subspace is subsequently obtained by applying one of the competing discriminant methods. The dimensionality of the discriminant subspace is determined by applying five-fold cross-validation on the training data within the range of [1,25]. Finally, performance is measured in the discriminant subspace using the average precision metric. Table I illustrates the performance over all classes of the MNIST dataset [20]. Since by using the entire training set performance saturates, we formed a reduced problem by using only the first 100 images of each class in the training set and we report performance on the entire test set. Table II illustrates the performance over all classes of the JAFFEE dataset [21]. We used the grayscale values of the facial images as a representation. For class-specific ranking problem, we applied five experiments by randomly splitting each class in 70%/30% subsets and using the first subset of each class to form the training set and the second ones to form the test set. Table III illustrates the performance over all classes of the 15 scene dataset [22]. We employed the deep features generated by average pooling over spatial dimension of the last convolution layer of VGG network [23] trained on ILSVRC2012 database. For each class-specific ranking problem, we applied five experiments by randomly splitting each class using 70%/30% splits.
In Tables I-III, the variants of the proposed method that outperform the corresponding baseline are highlighted for each problem. As can be seen, the performance obtained for the standard CSDA and its probabilistic version PCSDA (K = N n ) is very similar. The differences observed are mainly due to the small differences in the ranking of the used projection vectors (Eq. (27)). Moreover, exploiting subclass information of the negative class can be beneficial since performance improvements are observed in several cases. Comparing the CSSR in [14] and its probabilistic version PCSSR, it can be seen that the latter improves the performance considerably. This is due to the use of (27), which ranks all eigenvectors of CSSR according to their class-specific discrimination power.
Finally, in Table IV we compare the performance of CSDA and PCSDA for classification. Since each of the class-specific classification problems is imbalanced, we used the g-mean metric, which is defined as the square-root of the multiplication of the correct classification rate of each class. Classification is performed by using the classification rule g(·) in (33) for PCSDA. For CSDA a linear SVM is used is trained on vectors d i = |z i − µ|, similar to [5]. The value of C is optimized in the range 10 (−3:3) jointly with the discriminant subspace dimensionality by applying five-fold cross-validation on the training data. Comparing the performance of the CSDA+SVM classification scheme with that of PCSDA (K = N n ) on the MNIST and JAFFE datasets, we can see that the use of an additional classifier trained on the data representations in the discriminant subspace increases classification performance. By by allowing the negative class to form subclasses the performance increases for PCSDA and is competitive of that of the CSDA+SVM classification scheme. Overall, the PCSDA model achieves competitive performance without requiring the training of a new classifier in the discriminant subspace.  In this paper we proposed a probabilistic model for classspecific discriminant subspace learning that is able to incorporate subclass information naturally appearing in the negative class in the optimization criterion. The adoption of a probability-based optimization criterion for class-specific discriminant subspace learning leads to a classifier defined on the data representations in the discriminant subspace. We showed that the proposed approach includes as special cases existing class-specific discriminant analysis methods. Experimental results illustrated the performance of the proposed model, in comparison with that of related methods, in both verification and classification problems.