On the Relation Between ROC and CMC

We formulate a compact relation between the probabilistic Receiver Operating Characteristic (ROC) and the probabilistic Cumulative Match Characteristic (CMC) that predicts every entry of the probabilistic CMC as a functional on the probabilistic ROC. This result is shown to be valid for individual probabilistic ROCs and CMCs of single identities, based on the assumption that each identity has individual mated and nonmated Probabilitic Density Functions (PDF). Furthermore, it is shown that the relation still holds between the global probabilistic CMC of a gallery of identities and the average probabilistic ROC obtained by averaging the individual probabilistic ROCs of these identities involved over constant False Match Rates (FMR). We illustrate that the difference between individual probabilistic ROCs and the difference between global and average probabilistic ROCs provide an explanation for the discrepancies observed in the literature. The new formulation of the relation between probabilistic ROCs and CMCs allows us to prove that the probabilistic CMC plotted as a function of fractional rank, i.e., linearly compressed to a domain ranging from 0 to 1, will converge to the average probabilistic ROC when the gallery size increases. We illustrate our findings by experiments on synthetic and on face, fingerprint, and iris data.

On the Relation Between ROC and CMC Raymond N. J. Veldhuis , Senior Member, IEEE, and Kiran Raja , Senior Member, IEEE Abstract-We formulate a compact relation between the probabilistic Receiver Operating Characteristic (ROC) and the probabilistic Cumulative Match Characteristic (CMC) that predicts every entry of the probabilistic CMC as a functional on the probabilistic ROC.This result is shown to be valid for individual probabilistic ROCs and CMCs of single identities, based on the assumption that each identity has individual mated and nonmated Probabilitic Density Functions (PDF).Furthermore, it is shown that the relation still holds between the global probabilistic CMC of a gallery of identities and the average probabilistic ROC obtained by averaging the individual probabilistic ROCs of these identities involved over constant False Match Rates (FMR).We illustrate that the difference between individual probabilistic ROCs and the difference between global and average probabilistic ROCs provide an explanation for the discrepancies observed in the literature.The new formulation of the relation between probabilistic ROCs and CMCs allows us to prove that the probabilistic CMC plotted as a function of fractional rank, i.e., linearly compressed to a domain ranging from 0 to 1, will converge to the average probabilistic ROC when the gallery size increases.We illustrate our findings by experiments on synthetic and on face, fingerprint, and iris data.

I. INTRODUCTION
B IOMETRIC recognition systems proposed in the literature are often evaluated in the application contexts of biometric verification or biometric identification.The goal of biometric verification is to verify an identity claim based on two samples of biometric data, e.g., fingerprints, facial images, or iris scans.One of these samples, the reference, is usually stored in advance in a database, called the gallery, during a process called enrolment.The other sample, produced by the identity at verification time, is called the probe.The identity claim is accepted if the two samples are sufficiently similar, otherwise it is rejected.The goal of biometric identification is to determine the identity of an individual by comparing a probe to all the identity-labeled references stored in the gallery.If the identity of the individual is known to be represented in the gallery, we speak of closed-set identification.Otherwise, we speak of open-set identification and an additional verification step is needed to establish whether the found identity is indeed that of the individual.
Biometric verification and open-set identification are used in operational applications, respectively for example in biometric e-gates [1] and immigration search databases [2].Closed-set identification is not considered of operational value, because in operational biometric applications it can usually not be guaranteed that the probe is represented in the gallery [3].It remains, however, relevant, because it is often used in scientific publications to quantify the recognition performance of biometric solutions, see [4], [5], [6], [7] for some recent examples.
A relation between the ROC and CMC is plausible to exist, because both biometric verification and identification share a comparison module that quantifies the similarity of two samples of biometric data by a comparison score.This can be a similarity score, increasing with similarity of the biometric samples, or a dissimilarity score, decreasing with similarity.Both are equivalent, therefore our discussions will be in terms of similarity scores only.The comparison function can be used as a biometric comparator by thresholding the similarity score.If it is above a predefined threshold, the comparator decides for a match, i.e., the biometric samples originate from the same individual.Otherwise, it decides for a nonmatch.
In biometric verification, the comparator's decision is used to accept an identity claim in case of a match.Otherwise it is rejected.In biometric closed-set identification, similarity scores are computed by comparing a probe with all the references in the gallery.The identity label of the entry that produces the highest similarity score is taken as the identity of the individual.
A similarity score obtained by comparing biometric samples from the same identity is called, in ISO terminology [15], a mated similarity score.A similarity score obtained by comparing samples from different identities is called a nonmated similarity score.A good verification or identification performance requires mated similarity scores to be higher than nonmated similarity scores.However, due to the inevitable variability of biometric data the ranges of mated and nonmated similarity scores are likely to overlap, which will lead to comparison errors.In particular, a false match occurs if a nonmated similarity score is above the threshold and a false nonmatch occurs if a mated similarity score, is below the threshold.The corresponding error rates are the false-match rate (FMR), which is the fraction of false matches of all nonmated comparisons in a test, and the false-nonmatch rate (FNMR), which is the fraction of false nonmatches of all mated comparisons.The comparison threshold governs the trade-off between the FMR and the FNMR.Increasing it will result in a decreased FMR and an increased FNMR.In this paper it is more convenient to consider the true-match rate (TMR), which is fraction of correct matches obtained from all mated comparisons.It is the complement of the FNMR.The ROC plots the TMR as function of the FMR.The faster the ROC approaches 1 from the origin, the more discriminative the biometric comparison function is.
The typical error rates used in biometric verification are the false-acceptance rate (FAR) and the false-rejection rate (FRR).If failures to acquire can be ignored, a match decision will lead to acceptance of the identity claim.Then the FAR equals the FMR and the FRR equals the FNMR.Therefore, we will use FMR, FNMR, and ROC as performance characteristics for biometric verification.For a review of biometric testing terminology see [15] and [16].
The CMC is the commonly used performance indicator for biometric closet-set identification [8].It is conveniently explained as follows.After comparison of a probe with all references in the gallery, the r-best candidate list is defined as the list of references with the r highest similarity scores.For a closed-set identification experiment the CMC shows, for r ranging from 1 to the gallery size, the fraction of mated comparisons that are in the r-best candidate list.The CMC for r = 1 is the rank-1 recognition rate, which is often presented as a scalar performance indicator for a biometric identification system.A disadvantage of the CMC, in comparison with the ROC, is that it depends on the gallery size.This makes it meaningless to compare CMCs and rank-1 recognition rates for different gallery sizes or to present them when the gallery size is not specified.
The ROC and the CMC as explained above are empirical measures, obtained through experiments on identity-labeled biometric datasets.Alternatively, we can set up a probabilistic model for biometric recognition performance by assuming that the similarity score is a realisation of a random variable drawn from one of two probability-density functions (PDFs), i.e., the mated PDF for mated similarity scores and the nonmated PDF for nonmated similarity scores.Such a probabilistic model is useful, because it can be used to analyse and predict properties of empirical quantities and of the relations between them.In a probabilistic model the empirical ROC and CMC are seen as estimates of the underlying probabilistic ROC and CMC.
As our main result in this article: • We formulate a compact relation between the probabilistic ROC and the probabilistic CMC that, in our opinion, is more elegant than those presented earlier in [8], [9], [10].
It predicts every entry of the probabilistic CMC of a gallery of unique indentities as a functional on the probabilistic ROC (Refer Sections IV and V).This result is shown to be valid for individual probabilistic ROCs and CMCs of single identities, based on the assumption that each identity has individual mated and nonmated PDFs.• Furthermore, it is shown that the relation holds between the global probabilistic CMC of a gallery of unique identities and the average probabilistic ROC obtained by averaging the probabilistic ROCs of the identities in the gallery over constant FMRs (Sections V and IV).This average probabilistic ROC may differ from the global probabilistic ROC that is estimated in evaluation experiments and can be seen as the result of averaging individual ROCs over constant thresholds.• We conclude that the difference between the average probabilistic ROC and the global ROC provides an explanation for the discrepancies observed in the literature (Section IV).The new formulation of the relation between probabilistic ROCs and CMCs allows us to prove that the probabilistic CMC plotted as a function of fractional rank, i.e., linearly compressed to a domain ranging from 0 to 1, will converge to the average probabilistic ROC when the gallery size increases.• We illustrate our findings by experiments on real biometric data, namely the FRGC-V2 facial image dataset [17], the FVC2006 fingerprint dataset [18], and The Hong Kong Polytechnic University Cross-Spectral Iris Images Database [19].Individual ROCs are hard to measure accurately in experiments because of the limited number of mated similarity scores that can generally be computed from a dataset.We, therefore, also include experiments on synthetic data that mimics real biometric data.The remainder of this paper is organised as follows.
Section II discusses previous work on the relation between ROC and CMC and the discrepancies that have been observed between theoretical results and experimental observations.Section III reviews biometric comparison, verification and identification from a probabilistic point of view.This section introduces the mathematical concepts that are needed for what follows.The main theoretical result of this paper on the relation between ROC and CMC is presented in Section IV.In particular, this section proves that the individual probabilistic CMC for a known identity can be computed as an integral transformation of the individual probabilistic ROC and that the global CMC can be computed in a similar way from an average ROC, with the average taken for fixed false-match probabilities.Section V uses the main result of Section IV to prove convergence properties of the probabilistic CMC including that with increasing size of the gallery the CMC converges in shape to an equidistantly sampled version of the probabilistic ROC.In Section VI the theoretical results are validated by experiments on synthetic and real biometric data.Finally, Section VII presents conclusions.For legibility, all proofs have been deferred to appendices.

II. RELATED WORK
In [8,Appendix A.5] the authors develop an expression to predict the CMC.It is stated in terms of the probability density functions for mated and nonmated similarity scores and does not immediately reveal the relation with the ROC.
They find that their prediction improves if score normalisation is applied prior to search.This is further analysed and confirmed in [20].In [9] the authors extend the expression developed in [8] by including the probability that a mated similarity score is above a given threshold.Their result is explicitly based on the assumption that identities involved in the identification experiment set share the PDFs for mated and nonmated similarity scores, respectively.A relation of closed-set identification performance with the ROC is given in terms of the expected rank and the area under the ROC.This is rephrased in our Corollary 2. From their closed-set identification experiments, the authors conclude that the '[theoretical prediction] shows a prominent underestimation of the identification rate'.
In [10] the same prediction of the CMC as in [8] is presented, but without the restriction on the threshold.The assumption that all identities share the same PDFs or mated and nonmated similarity scores, respectively, is not explicitly made.A further approximation based on the assumption that PDF of the mated similarity scores is narrow compared with that of the nonmated similarity scores leads to a simpler expression.The authors illustrate that their theoretical predictions underestimate the empirical results through a set of experiments.
In [11], [12] it is concluded based on experimental data that it was needed to present both identification and verification performance when testing the performance of a biometric comparison system.This confirms that, although the relation between CMC and ROC such as presented in [8], [9], and [10] appears to be mathematically correct, it cannot be used to predict a CMC from a measured ROC.This is also confirmed by [13] in which examples from combinations of good experimental CMCs and poor ROCs were presented.In [14] it was found that splitting the data into 'goats', 'sheep', and 'lambs' according to Doddington's menagerie [21], resulted in distinct CMCs for each category, while they share the same ROC.
The findings reported in [8], [9], [10], [11], [12], [13], [14] are summarised well in [14] by the statement 'it is clear that support in the literature for a direct relationship between the ROC and CMC curves is mixed.'With this paper we hope to resolve the observed discrepancies between the CMC are predicted from the ROC and experimental results.

III. BIOMETRIC COMPARISON, VERIFICATION AND CLOSED-SET IDENTIFICATION
This section reviews biometric comparison from a probabilistic point of view.We assume that biometric data is random and hence similarity scores are random variables with distinct underlying PDFs for the mated and nonmated similarity scores.We also assume that these PDFs are individual.This will lead to the introduction of individual probabilistic ROCs and probabilistic CMCs.This is mainly of theoretical importance, as in most practical evaluations of biometric systems individual PDFs cannot be estimated reliably due to limited numbers of mated similarity scores.Therefore, the performance of biometric systems is commonly expressed in terms of the global, aggregated, ROCs and CMCs, and other performance indicators are derived from those.Moreover, global results are better predictions of future performance, because future users are usually unknown.
Let the set x denote a probe, and the set y a reference, either for verification or closed-set identification.We use a set notation to be as general as possible.For instance, x and y may contain different numbers of biometric features or heterogeneous features such as from low-and high-resolution facial images.The only restriction we put on x and y is that they allow for the computation of a similarity score s = s(x , y) ∈ S ⊂ R, where S denotes the set of possible similarity score outcomes and R the set of real numbers.In our probabilistic approach, probe and reference are random.Hence, s is a random variable with a PDF depending on who produced x and y.For x and y produced by identities ω and ν, respectively, this PDF is denoted by φ(s|ω, ν).
Biometric performance characteristics are defined for a specific population P where probe identity ω and reference identity ν are drawn from.We define the mated similarity score PDF of an arbitrary probe identity ω ∈ P by and the nonmated similarity score PDF of the same probe identity by with |P| size of the population P.This represents the PDF of the nonmated similarity score of a probe from ω compared with a reference from an unknown random identity.The global mated and nonmated similarity score PDFs over the population, representing similarity score PDFs from unknown random probe and reference identities, are then given by, respectively, The biometric comparator compares s = s(x , y) to a threshold t to determine whether x and y originate from the same identity or not.The false-match probability p fm (t|ω) and truematch probability p tm (t|ω) of identity ω at similarity score threshold t are defined by respectively.The global false-match probability p fm (t) and the true-match probability p tm (t) are then given by Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
respectively.Note that the false-and true-match probabilities are properties of the similarity score taken as a random variable, while the FMR and TMR are empirical entities measured from realisations of the random similarity scores and can be seen as estimates for the false-and true-match probabilities, respectively.
The probabilistic ROC of a biometric comparison function is defined as the true-match probability presented as a function of the false-match probability.For an identity ω we define the individual probabilistic ROC by The symbol α will be used for the false-match probability as the dependent variable in a probabilistic ROC.The global probabilistic ROC is defined by The empirical ROC that is obtained in an evaluation experiment can be seen as an estimate for the global probabilistic ROC.
The average probabilistic ROC, obtained by (vertically) averaging individual probabilistic ROCs over a constant falsematch probability will also be needed in Section IV.It is defined by Note that in general ρ(α) = ρ(α).However, the following property describes an important case for which ρ(α) = ρ(α): The proof is given in Appendix A. Vertical averaging of ROCs is discussed in [22].
In closed-set identification, a gallery G of size n is a set of references y i , i = 1, . . ., n, with corresponding identities ω i , i = 1, . . ., n, i.e., where each reference originates from a different identity, randomly drawn without replacement from the population P.
The r-best list of a probe x contains the identities of the r pairs {ω i , y i } ∈ G with the largest similarity scores s(x , y i ).
In closed-set identification, the individual probabilistic CMC denoted by cmc(r|ω; n) quantifies for r = 1, . . ., n the probabilities that ω is in the r-best list of a gallery of size n, containing {ω, y} and n − 1 randomly drawn pairs {ω j , y j }, then quantifies the probabilities that any probe represented in a randomly drawn gallery is in the r-best list.

IV. RELATING CMC AND ROC
As our main result, we present a compact relation between the probabilistic CMC and the probabilistic ROC, both for individual and global CMCs.For this, we need the β-PDF β(x; a, b) [23] with parameters a ∈ N and b ∈ N: In Bayesian statistics the β-PDF is used to model probability parameters such as that of the binomial distribution [23].Here it appears naturally in the derivation of our main result.Its expectation and variance are given in Section V. Examples of β-PDFs are shown in Figure 1.
The following theorem expresses the probabilistic CMC as a functional on the probabilistic ROC with the β-PDF as a kernel.The proof is given in Appendix B.
Theorem 1: Given (a) a gallery G of size n, with one entry per identity, containing the entry {ω, y} for a probe identity ω and n − 1 other entries randomly drawn from the population of identities, and (b) a biometric comparison system for identity ω is characterised by a probabilistic ROC ρ(α|ω).Then the individual probabilistic CMC cmc(r|ω; n) and the global probabilistic CMC cmc(r; n) are given by, respectively, with ρ(α) defined in Equation (11). Figure 1 plots examples of the kernels β(α; r, n − r) in Equation (16) and Equation ( 17) as a function of α for (r, n) = (4, 40), (10,40), (40, 400), (100, 400).These kernels act as local weighing functions on the probabilistic ROC, which provides more insight in the relation between the probabilistic CMC and ROC.We observe that the two graphs (r, n) = (4, 40) and (r, n) = (40, 400), with fractional rank r n = 1  10 as well as for the two graphs (r, n) = (10, 40) and (r, n) = (100, 400), with fractional rank r n = 1 4 , the graphs remain more or less in the same position, but become narrower with increasing gallery size n.This convergence behaviour will be further discussed in Section V.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
Summarising, we conclude that Equation (16) shows, in a compact form, that the individual probabilistic ROC contains all the information needed to compute an individual probabilistic CMC.Expression (17) gives a compact relation between the global probabilistic CMC that is often estimated as a performance indicator in experiments, and the average probabilistic ROC ρ(α).The predictions of the CMCs in [8], [9], [10] are based on applying Equation (33) (in Appendix) to experimental FMRs and TMRs that are obtained by averaging over fixed thresholds, which is common in biometric evaluations.However, this would correspond to using the global ROC ρ(α) in Equation (17) for prediction of the CMC, instead of ρ(α).The global ROC ρ(α), as we will also show in Section VI, may differ from ρ(α) and this, in our view, explains the discrepancies between predicted and measured CMCs that have been observed in [8], [9], [10], [11], [12], [13], [14].
In [8], [20] it was shown that individual score normalisation improved the prediction of the CMC.This can be explained as follows.As a result of individual score normalisation, all identities will have similar FMRs.Then, according to Equation ( 12), the global ROC will become more similar to the average ROCs well and the prediction of the CMC will become better.

V. CONVERGENCE PROPERTIES
We observed in Figure 1 that for a constant fractional rank the kernels maintain their positions on the α-axis, but become narrower.This behaviour is easily explained by the expectation and variance of the β-PDF β(x; r, n − r), which are given by, respectively, Informally speaking, one can say that for increasing n and a fixed fractional rank r n , β(α; r, n − r) will converge to a Dirac impulse function δ(α − r n ).Therefore, for large n and fixed r n , the integral in Equation ( 16) effectively comes down to sampling the ρ(α|ω) at α = r n , which implies that the shape of the individual CMC as a function of fractional rank converges to the individual ROC for large galleries.This is illustrated by Figure 2, which plots cmc(r|ω; n) for n = 30, 60, 120 as a function of fractional rank r n for ρ(α|ω) = (1 − exp(−50α))/(1 − exp(−50)).Formally, we have the following theorem.
Theorem 2: Given (a) a gallery G of size n, with one entry per identity, containing the entry {ω, y} for a probe identity ω and n − 1 other entries randomly drawn from the population of identities, and (b) a biometric comparison system that for identity ω is characterised by a probabilistic ROC ρ(α|ω).Then the individual probabilistic CMC cmc(r|ω; n) and the global probabilistic CMC cmc(r; n) are given by, respectively, In Equation ( 20) O( 1 n ) is the Landau symbol that bounds the rate of convergence [24].
The proof is given in Appendix D.
From Theorem 2 it follows that for large galleries the probabilistic CMC indeed converges at a rate 1/n to the (average) probabilistic ROC.Note that the last but one line of Equation (43) in the proof implies that: (a) the rate of convergence is close to 1/n 2 for ranks close to 1; (b) the rate of convergence decreases where the bend in the ROC is sharpest, because there the second derivative is largest; and (c) the approximation becomes perfect when the ROC flattens off, because there the second derivative vanishes.The convergence behaviour is summarised in the following corollary.
Corollary 1: Given (a) a gallery G of size n, with one entry per identity, containing the entry {ω, y} for a probe identity ω and n − 1 other entries randomly drawn from the population of identities, and (b) a biometric comparison system that for identity ω is characterised by a probabilistic ROC ρ(α|ω).Then the individual probabilistic CMC cmc(r|ω; n) and the global probabilistic CMC cmc(r; n) satisfy, respectively, The proof follows on taking the limit n → ∞ in Equation (22) and Equation (23).
In Figure 2, we observe that the probabilistic CMC approximates the probabilistic ROC from below.This is generally true for individual and average probabilistic ROCs that are concave, which is a desired property for an ROC.
The area under the ROC, i.e., 1 0 ρ(α|ω)dα or 1 0 ρ(α)dα, is often used to quantify the performance of a biometric comparison system [25], [26].The closer it is to 1, the more discriminative the system is.For a CMC plotted as a function of fractional rank, a similar property is desirable.The curve should be close to the point (0, 1) in the ( r n , cmc)-plane, which will be achieved if the area under the CMC, defined Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
as 1 n n r=1 cmc(r|ω; n) is close to 1.The following corollary relates the area under the ROC with the that under the CMC, showing that for large galleries both requirements are equivalent.The proof is in Appendix E. In a somewhat different form this result is also presented in [9,Eq. 22].
Corollary 2: Given (a) a gallery G of size n, with one entry per identity, containing the entry {ω, y} for a probe identity ω and n − 1 other entries randomly drawn from the population of identities, and (b) a biometric comparison system that for identity ω is characterised by a probabilistic ROC ρ(α|ω).Then the average individual probabilistic CMC cmc(r|ω; n) and the average global probabilistic CMC cmc(r; n) are given by, respectively, 1 Hence, So far, theoretical relations between the probabilistic ROC and CMC have been derived.In Section VI these results will be illustrated by experiments that produce empirical ROCs and CMCs.

VI. ILLUSTRATIVE EXPERIMENTS
The goal of the experiments is to illustrate the validity of Theorems 1 and 2. In order to do that, we will estimate empirical ROCs and CMCs from data, use Equation ( 16) and Equation (17) to predict CMCs form them, and compare these with the empirical CMCs.
We will first present experiments based on synthetically generated random biometric samples.In this way we generate sufficient samples to produce accurate ROCs and CMCs.Second, we will present experiments on real data, namely the FRGC-V2 facial image dataset [17], using samples from identities that contributed a larger number of 20 samples, and the FVC2006 fingerprint dataset [18] and The Hong Kong Polytechnic University (HKPU) Cross-Spectral Iris Images Database [19], both using samples from identities that contributed 10 samples.
In the previous sections we worked with probabilistic ROCs and CMCs, which follow from the underlying probability density functions of similarity scores.In the current section, we will work with empirical ROCs and CMCs and, formally, we should introduce new notation for that.In order to avoid notational complexity, we will not do that, except for the CMCs that are predicted from the empirical ROCs, which we will denote as cmc(r|ω; n) and cmc(r; n).
We also use the similarity scores to compute individual empirical CMCs cmc(r|ω; n) for all identities ω.We do this by mimicking a large number of identifications.For identity ω and every sample index i, n − 1 nonmated similarity scores s(x ω,i , x ν,j ν ), ν = 1, . . ., m, ν = ω, with each j ν randomly drawn from {1, . . ., m} are selected.Then, for m − 1 mated similarity scores s(x ω,i , x ω,j ), j = 1, . . ., m, j = i, the set {s(x ω,i , x ω,j ), {s(x ω,i , x ν,j ν )} ν =ω } is sorted in descending order and the ranks of the mated similarity score s(x ω,i , x ω,j ) are collected in a histogram with bins 1, . . ., n.This mimics identification in a gallery of size n with a mated enrolled feature vector x ω,i and n − 1 nonmated enrolled feature vectors x ν,j ν , ν = 1, . . ., m, ν = ω with probes x ω,j , j =, . . ., m, j = i.In order to increase the accuracy of the individual empirical CMC this is done m g times for every sample index i of identity ω by randomly drawing m g sets {s(x ω,i , x ν,j ν )} ν =ω of nonmated similarity scores.Each time the rank histogram is updated.Per identity ω a CMC is computed from the histogram, based on m g m(m − 1) rankings of similarity scores.The global empirical CMC cmc(r; n) is computed by averaging the individual CMCs cmc(r|ω; n).The experimental parameters n (gallery size), m (the number of features per identity), and m g (the number of randomly drawn galleries per mated gallery entry) are specified in Table I.
We will present the results in three types of graphs:

A. Synthetic Data
We simulate a simplified biometric feature comparison, based on Gaussian between-and within-identity variations.Let x = b+w be a feature vector of an identity, with b ∼ N(0, B ) the individual feature mean and w ∼ N(0, W ) modelling the within-identity variations.In our synthetic-data experiments x ∈ R 4 and B is a diagonal covariance matrix with diagonal elements σ 2  1 , . . ., σ 2 4 = (0.5, 0.8, 0.85, 0.9).
Individual empirical ROCs, individual predicted CMCs, individual empirical CMCs, the average individual empirical ROC, the empirical ROC, and the empirical CMC are computed for gallery sizes n = 10 and n = 100 and for 2 classifiers: the log-likelihood-ratio classifier optimised for the data and the cosine-similarity classifier.For each combination of gallery size and classifier a new data set is randomly generated.The experimental parameters are given in Table I.
The log-likelihood-ratio classifier computes a similarity score with D = 4 and σ 2 d specified in Equation (30).The cosinesimilarity classifier computes a similarity score  also shows that the empirical ROC ρ(α) and the average individual empirical ROC ρ(α|ω) are different.Figure 3(c) directly compares the empirical CMC with the predicted CMC, showing that the deviation is largest at lower rank values.
Figures 4(a), 4(b), and 4(c), show the results for gallery size n = 10 with the cosine-similarity classifier.The observations are mainly similar to those made for gallery size n = 10 with the log-likelihood-ratio classifier.The main difference is that here the empirical ROC ρ(α) and the average individual empirical ROC ρ(α|ω) in Figure 4(b) are more similar for the cosine-similarity classifier.
Figures 5(a), 5(b), and 5(c), show the results for gallery size n = 100 with the log-likelihood-ratio classifier.Here we can see that both the individual as well as the global empirical and predicted CMCs are closer together than in the n = 10 experiments.In fact, from Figure 5(c) we see that the predicted CMC approximates the empirical quite well.We can also observe that the empirical and predicted CMCs, plotted as functions of r/n more closely approximate the average ROCs as in the n = 10 experiments as was proved in Theorem 2. The empirical ROC ρ(α) is above the average individual empirical ROC ρ(α), which is to be expected, because the log-likelihoodratio classifier is the optimal classifier and the best recognition performance is obtained when the same log-likelihood-ratio threshold is used for all identities as was shown in [27].The average ROC is obtained by setting the threshold such that all identities have the same false-match rate, which leads to a different log-likelihood-ratio threshold for each identity.This effect was also visible for the n = 10 case, but not for the entire ROC.We expect this observation because ROCs obtained with only 10 identities are less accurate than ROCs obtained with 100 identities.Figures 6(a), 6(b), and 6(c), show the results for gallery size n = 100 with the cosine-similarity classifier.Again, the observations are mainly similar to those made for gallery size n = 100 with the log-likelihood-ratio classifier.The main difference is again that here the empirical ROC ρ(α) and the average individual empirical ROC ρ(α|ω) in Figure 6(b) are more similar for the cosine-similarity classifier.In fact, they seem to overlap completely.We assume that this is because in the given constellation of feature vectors x ∼ N(0, I) and the cosine-similarity classifier, we have the nonmated similarity scores that are equally distributed, i.e., φ nm (s|ω) = φ nm (s), then according to Equation ( 12) the average ROC coincides with the global ROC.
The above experiments confirm our main result, laid down in Theorem 1, that the CMC can be predicted from the average ROC.For the smaller gallery size there are minor deviations at the lower rank values, that can be ascribed to inaccuracies of individual ROCs obtained with few samples.

B. Face Recognition Data
As our primary goal is illustrating the relation between the ROC and the CMC, we choose the FRGC-V2 face dataset [17], where the performance is not near-ideal.Though one can use LFW and alike datasets, we note that most of the newer face recognition systems provide near-ideal performance [28] on those, deeming them less useful for our purposes.The FRGC-V2 dataset consists of a training split of 222 unique identities and a testing split of 466 unique identities.We employ the testing split from which we preselect the 285 identities with at least 20 images.From these identities we randomly retain 20 images per identity.We further preprocess the images by detecting the face region with a tight cropping [29].We extract features x ω,j ∈ R 512 and compute all the relevant similarity scores s(x ω,j , x ν,l ), (ω, j) = (ν, l), using the pre-trained ElasticFace [30] face recognition system based on deep learning to extract the feature vectors.These are compared using the cosine-similarity classifier in Equation (32).The employed pre-trained models are trained on the MS1MV2 dataset [28] and validated on various datasets such as LFW, AgeDb-30, CPLFW, CALFW and CFP-FP for state-of-the-art performance.
For gallery sizes n = 10 and n = 100, we randomly selected, respectively, 10 and 100 identities out of the preselected 285.In this way we have the same maximum numbers of identities and gallery sizes as in the synthetic-data experiments described in Subsection VI-A.The experimental parameters are specified in Table I.   showing that the deviation is largest at lower rank values where the predicted CMC underestimates the empirical one.This was not the case for the synthetic data and also not for the face recognition data experiments with gallery size n = 10.We expect this for lower values of α as the individual and average ROCs are much steeper for the face recognition data experiments than for those on synthetic data.Since only few data points are available for lower values of α, the concave ROCs are systematically underestimated when Equation ( 16) is applied to the piecewise linear approximation of the ROCs, leading to underestimation of the predicted CMCs.For gallery size n = 10, the ROC is less steep, and more data points are available, therefore the piecewise linear approximation of the ROC is more accurate.
As was the case with the synthetic-data experiments, the experiments in this subsection confirm our main result, laid down in Theorem 1, that the CMC can be predicted from the average ROC.In steeper parts of the ROC there are deviations in the prediction that can be ascribed to the combination of the availability of fewer datapoints and the piecewise linear approximation of the individual and average ROCs.

C. Fingerprint Recognition Data
We employ set A of the DB2 subset of the Fingerprint Verification Competition 2006 [18].This dataset was captured using an optical sensor and consists of 150 unique fingerprints with 12 samples per finger.The fingerprints correspond to images of size 400x560 represented in 256 gray levels in uncompressed format with a 569 dpi resolution.We choose a subset of 100 randomly selected fingerprints with 10 samples for each fingerprint.For each of the fingerprints, we extract minutiae using MINDTCT and compare the minutiae using Bozorth3 [31].
Because only 10 fingerprints per individual are available, there are only 45 mated comparisons to compute the individual empirical ROCs, which will result in empirical ROCs with a high granularity in the vertical coordinates.This will make the computation of individual predicted CMCs inaccurate.Therefore, we will only present the global empirical ROC, the average individual empirical ROC, the predicted global CMC, and the empirical CMC, for gallery size n = 100.For this larger gallery size, the inaccuracies in the individual empirical ROCs and CMCs average out in the global empirical ROCs and CMCs respectively.
Figure 9(a) shows the empirical ROC, the average individual ROC, the global predicted CMC cmc(r; n) and the global empirical CMC cmc(r; n), which are all close.This is to be expected because of the higher recognition performance.Figure 9(a) directly compares the empirical CMC with the predicted CMC, showing that at lower rank values the predicted CMC underestimates the empirical one somewhat, whereas at higher ranks it overestimates.We expect that, as was also mentioned in Section VI-B, that the underestimation at lower rank values is due to the steepness of this ROC.Note the scale of the axis, from which we can see that the prediction is close.Here, there is no underestimation of the CMC at lower rank Fingerprint recognition data experiments.Gallery size n = 100, FVC2006 data [18], fingerprint recognition system: MINDTCT with Bozorth3 [31].
values, which we ascribe to that this ROC is less steep than those for face and fingerprint recognition.

D. Iris Recognition Data
We use Near-Infra Red (NIR) iris image subset from The Hong Kong Polytechnic University Cross-Spectral Images Database [19].This database consists total 12,540 iris images collected from left and right eyes of 209 subjects in two different spectrums (Visible and NIR) with 15 different instances.For the sake of our experiments, we consider a subset of images from 100 randomly selected subjects with 10 images from left eye each captured in NIR spectrum.For all the iris images, we segment and localize iris region following which we use Daugman's rubber sheet model [32], [33].Further, we extract binarised 1D Log-Gabor codes from iris images as features and employ the fractional Hamming distance to compare two iris codes Formally, the fractional Hamming distance is a rational number and not a real number.However, the mated and nonmated probabilities can be approximated by continuous PDFs, which allows us to apply our results.
For the same reason as given for the fingerprint data in Section VI-C we only present the global empirical ROC, the average individual empirical ROC, the predicted global CMC, and the empirical CMC, for gallery size n = 100.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.Figure 10(a) shows the empirical ROC, the average individual ROC, the global predicted CMC cmc(r; n) and the global empirical CMC cmc(r; n).We observe that the empirical ROC deviates somewhat from the other graphs, which are closer to each other.We can also see that the recognition performance is poor, with an equal error rate of about 15%. Figure 10(b) directly compares the empirical CMC with the predicted CMC, showing that the prediction is close.

VII. CONCLUSION
We extended earlier experimental and theoretical work [8], [9], [10], [11], [12], [13], [14] on the relation between the ROC and the CMC.Our main result is that, in a probabilistic framework, an individual's CMC can be predicted from the individual's ROC as a functional with a kernel based on the β-PDF.This result cannot be applied directly to verification and closed-set identification experiments to provide a relation between a global CMC and a global ROC for two reasons.The first is that the experimental ROC is based on counting true and false matches with similarity scores above fixed thresholds, while the established relation holds for ROCs obtained by averaging individual ROCs at fixed false-match rates.The second reason is that, if one should decide to compute individual ROCs to predict a CMC from, this would require a large number of mated similarity scores for each identity in the gallery, to have the empirical ROC approximate the probabilistic one well enough.This is particularly true for the steeper parts of the ROC, determined by the tails of the similarity score distributions.The main result is complemented with convergence properties for increasing gallery size.In particular, we show that for increasing gallery size, the individual CMCs as a function of fractional rank, converge to the individual ROCs.Our results provide explanations for discrepancies related to the ROC and CMC observed in the literature.
The theory is illustrated by experiments on synthetic data with Gaussian within-and between-class variations, and on real data, namely the FRGC-V2 facial image dataset [17], the FVC2006 fingerprint dataset [18], and The Hong Kong Polytechnic University Cross-Spectral Iris Images Database [19].The experiments confirm the theoretical results.It must be noted that the real-data experiments show that the prediction of the CMC from individual ROCs is less accurate where the ROC is steeper, due to less accurate estimation of the ROCs in the tails of the similarity score distributions.
Finally, because we found that there is a probabilitytheoretical relation between the individual ROCs averaged over fixed false-match rates and the global CMC, but generally not between the global ROC and CMC, we cannot conclude that experimental biometric results only need to be presented as ROCs and that CMCs would not have added value.However, we think that presenting ROCs has the advantage over presenting CMCs that an ROC is independent of the gallery size, and therefore lends itself better for a general comparison of biometric recognition performance.
with 0 ≤ ξ ≤ 1.Note that ξ depends on α.On substitution of Equation (42) into 1 0 ρ(α|ω)β(α; r, n − r)dα, the first term of the right-hand side of Equation (42) remains unchanged, because it is a constant.The second vanishes because E{α} = r n , cf.Equation (18), and the remainder term cannot be computed analytically, but with Equation (19) we have This completes the proof.

APPENDIX E PROOF OF COROLLARY 2
Proof: We will only prove the individual case in Equation (24).The global case follows by averaging both sides of Equation (24) over the probe population.Starting from Equation ( 16) we find that

Fig. 2 .
Fig. 2. Plots of ρ(α|ω) as a function of α and cmc(r|ω; n) as a function of fractional rank r n for n = 30, 60, 120, illustrating how the individual CMC converges to the individual ROC.
Figures 3(a), 3(b), and 3(c) show the results for gallery size n = 10 with the log-likelihood-ratio classifier.Figure 3(a) shows 3 of the 10 individual empirical ROCs ρ(α|ω) and the corresponding identity predicted CMCs cmc(r|ω; n) and the individual empirical CMCs cmc(r|ω; n).We see that there can be quite a spread between the individual empirical ROCs and that the individual predicted CMCs deviate slightly from the individual empirical CMCs, but this deviation is reduced if we consider the global predicted CMC cmc(r; n) and the global empirical CMC cmc(r; n) in Figure 3(b).This figure

Figures 7 (
a), 7(b), and 7(c), show the results for gallery size n = 10.The results are similar to those obtained with synthetic data and gallery size n = 10. Figure 7(a) shows 3 from the 10 individual empirical ROCs ρ(α|ω) and the corresponding individual predicted CMCs cmc(r|ω; n) and the individual empirical CMCs cmc(r|ω; n).As was the case with the synthetic data, there is spread between the individual empirical ROCs.The individual predicted CMCs deviate slightly from

Proof:
We start with the expression for the CMC cmc(r|ω; n) = S r l=1 n − 1 l − 1 p l−1 fm (s|ω)