Privacy-Preserving Multi-Biometric Indexing Based on Frequent Binary Patterns

The development of large-scale identification systems that ensure the privacy protection of enrolled subjects represents a major challenge. Biometric deployments that provide interoperability and usability by including efficient multi-biometric solutions are a recent requirement. In the context of privacy protection, several template protection schemes have been proposed in the past. However, these schemes seem inadequate for indexing (workload reduction) in biometric identification systems. More specifically, they have been used in identification systems that perform exhaustive searches, leading to a degradation of computational efficiency. To overcome these limitations, we present an efficient privacy-preserving multi-biometric identification system that retrieves protected deep cancelable templates and is agnostic with respect to biometric characteristics and biometric template protection schemes. To this end, a multi-biometric binning scheme is designed to exploit the low intra-class variation properties contained in the frequent binary patterns extracted from different types of biometric characteristics. Experimental results reported on publicly available databases using state-of-the-art Deep Neural Network (DNN)-based embedding extractors show that the protected multi-biometric identification system can reduce the computational workload to approximately 57% (indexing up to three types of biometric characteristics) and 53% (indexing up to two types of biometric characteristics), while simultaneously improving the biometric performance of the baseline biometric system at the high-security thresholds. Code is available at https://github.com/dosorior/FBP-Multi-biometric-Indexing.


I. INTRODUCTION
Biometric technologies are rapidly gaining popularity due to their wide applicability.Biometric recognition of individuals based on distinctive biometric characteristics (BCs), e.g.face or iris, is successfully deployed in many personal, commercial, and governmental identity management systems around the world, e.g.border control, and national ID systems.A report on the global biometric market concerns the annual growth rate in biometric technologies by estimating 45.96 billion dollars in 2024 [1].In addition, biometrics vendors demand interoperability and deployment assuring maximum usability by including multimodal biometric solutions, e.g.fight against fraud in banks [2] and border and immigration [3] processes.These requirements (i.e.interoperability and usability) motivate the development of biometric characteristic-agnostic systems.In particular, solutions that operate a common feature space while preserving high biometric performance by enabling new fusion schemes prior to any processing step in a biometric system.From an efficiency perspective, existing large-scale biometric systems are processing millions of subjects in the enrolment (e.g.[4]) and re-enrolment processes (e.g.[5]), respectively.
The above facts show the increase in computational cost.Also, the growth in monetary costs as large companies accelerate their large-scale processing by investing in advanced technologies (e.g.hardware and speed-up devices).The challenging identification and duplicate enrolment check scenario where generally an exhaustive search (i.e.one-to-many comparison) is a time-consuming task, demands practical solutions which are not dominated by the number of comparisons and hence a high computational workload.In recent years, significant interest has been raised in addressing this topic by investigating the workload reduction (WR) methods [6], e.g.biometric indexing schemes, which have been introduced as methods with the aim of processing large amounts of biometric data with reasonable transaction times.
In addition to the emerging topic of accelerating searches within large-scale biometric databases, the violation of data privacy came as a shock for many individuals (e.g.period tracker scandal [7]) as sensitive information (e.g.personal health data) could be fully exposed.That is, in the context of a biometric system, unprotected storage of biometric references could lead to different privacy threats such as identity theft, linking across databases, or limited renewability [8].Also, privacy regulations, e.g. the European Union (EU) General Data Protection Regulation 2016/679 (GDPR) [9], usually define biometric information as sensitive data which requires strong mechanisms for the protection of stored data.
In the context of privacy protection, privacy-preserving biometric solutions have been challenged by natural intraclass variance of different biometric characteristics.Conventional cryptographic methods would require decryption of protected biometric data prior to the comparison step in order to prevent the effect of biometric variance in the encrypted domain.This is not the case with biometric template protection schemes [10], [11] which enable a comparison of biometric data in the transformed domain (encrypted) and hence a permanent protection of biometric data.They are usually distinguished in the literature as cancelable biometrics and biometric cryptosystems.Generally, the latter category is not suggested in identification scenarios (where the workload is dominated by the typical exhaustive search-based), as they require complex comparison methods (e.g.[12], [13]), in contrast to cancelable schemes (e.g.[14]).
Recently, Osorio-Roig et al. [15] introduced the proof-ofconcept of frequent binary patterns for indexing deep cancelable face templates.This privacy-preserving solution allowed working on different cancelable protection schemes (e.g.socalled BioHashing [16] and variants of Index-of-Maximum Hashing [17]) ensuring a trade-off between computational workload and biometric performance for protected biometric identification systems.Motivated by our previous study (see [15]), we present in this work (to the best of the authors' knowledge) the first privacy-preserving multi-biometric identification system based on the search of frequent binary patterns over cancelable biometric templates.The main contributions of the article are: • An overview that delves into the area of computational workload reduction for the indexing of protected biometric templates in identification systems based on a single biometric characteristic.• The successful application of the proof-of-concept of frequent binary patterns on individual biometric characteristics, i.e. face, iris, and fingerprint.• An efficient privacy-preserving multi-biometric system that is agnostic across cancelable biometric template protection schemes (with binary representation) and biometric characteristics.This solution is able to operate on the most secure processing step (i.e.feature level) in a biometric system by enabling fusion strategies on the concept of frequent binary patterns at two steps: the representation-and feature-based step.The fusion in the representation-step retrieval and indexing shows that the workload reduction and the biometric performance are irrespective of the ranking (i.e.order of priority) of the biometric characteristics, in contrast to the fusion in the feature-step retrieval and indexing.• A thorough theoretical and empirical analysis of the trade-off between computational workload reduction and biometric performance of the proposed identification system on multi-modal large-scale datasets with state-of-theart biometric recognition systems.Experimental evaluations compliant with the metrics defined in the ISO/IEC 19795-1:2021 [18] show that a protected multi-biometric identification system can reduce the computational workload to approximately 57% (indexing up to three types of biometric characteristics) and 53% (indexing up to two types of biometric characteristics), while simultaneously improving the biometric performance at the high-security thresholds of a baseline biometric system.
The remainder of this work is organised as follows: related works summarising concepts related to information fusion, workload reduction and biometric template protection are revisited in Sect.II.In Sect.III, the proposed system is described in detail.Sect.IV presents the experimental evalua-tions and results are reported and discussed in Sect V. Finally, conclusions are drawn in Sect.VI.

II. RELATED WORKS
This section describes the background and related work on reducing computational workload in protected biometric identification systems.Whereas Sect.II-A introduces the fusion strategies commonly used in biometrics, Sect.II-B addresses the problem of workload reduction on biometric systems.Finally, key work related to the workload-reduction and biometric identification systems areas on biometric template protection is summarised in Sect.II-C.

A. Biometric Information fusion
Biometric information fusion allows combining biometric data at different levels of processing in a biometric system.Those systems which enable biometric information fusion are known in the literature as multi-biometric systems.Generally, multi-biometric schemes combine or fuse multiple sources of information to improve the overall discriminative power of a single biometric recognition system [19].The fusion strategies can be categorised in the biometric context as multi-types, multi-sensorial, multi-algorithms, multi-instances, and multipresentations [20], [21].
The system proposed in this work relates to the first scenario, i.e. multi-type, which relies on the fusion of different types of BCs (e.g.facial and iris images).Specifically, three types of BCs are selected and subsequently utilised in a binning and fusion scheme.Note that given the simplicity of the proposed scheme, other fusion categories, such as multisensorial, multi-instances and multi-presentations can be also employed.In addition to the general categories above, several levels of biometric processing can be distinguished at which information fusion can take place [20], [21]: sensor, feature, score, rank, and decision.
In the scope of this article, the fusion of information from multiple features and at the score level is of major interest, as the proposed scheme in Sect.III is designed to operate at those levels of the biometric processing pipeline.The featurelevel fusion has been also considered, as it is among the most convenient techniques contributing to the highest privacy protection and security level, respectively [22], [10].
Information fusion in biometrics has been widely addressed in the scientific literature.An interested reader is therefore referred to, e.g.Ross et al. [20] for a general introduction to this topic and Paul et al. [23] for score-level fusion specifically, as well as Dinca and Hancke [24], Singh et al. [25], and ISO/IEC TR 24722 [21] for more recent works relating to the general topic of biometric information fusion.

B. Computational workload reduction
Biometric identification systems require fast response times, as the typical exhaustive search-based retrieval method demands high computational costs.Thus, the computational complexity tends to grow linearly with the number of enrolled data subjects [26].As expected, the investment in expensive hardware that contributes to the parallel processing/distribution can be used to maintain a quick response time in a biometric identification transaction.Whereas many companies spend high monetary costs to achieve the desired times, one possibility that is often overlooked is the optimisation of the underlying software and/or algorithms.In this context, a solution to said problems (i.e.high computational and monetary costs) is the research field of computational workload reduction which allows decreasing the dependence on the investment of the physical infrastructure and focusing more attention on the software and/or algorithms.Workload reduction-based methods work directly on the optimisation of the amount of computations required for some specific tasks in the biometric processing pipeline.For instance, for a biometric identification transaction, the computational costs at the biometric template comparison level typically dominate the computational effort of the entire system.Thus, most of these methods have been categorised in [6] as pre-selection approaches.These methods seek to reduce the number of biometric template comparisons (i.e.reducing the search space (see e.g.[27])), and feature transformation, aimed at accelerating the computational cost produced in a one-to-one comparison (see e.g.[28]).The former is of interest in the context of this article.For further information on such methods, the reader can be referred to [6].
Naturally, those workload reduction-based techniques (i.e.pre-selection methods) have achieved decreasing the search spaces w.r.t. the typical exhaustive searches.Conceptually, such approaches are mostly custom-built for specific biometric systems, e.g.single biometric characteristics or feature extractors introducing specific representations, and are not expected to be applicable within other systems, e.g. containing different types of biometric characteristics to be processed.In addition, they are primarily designed to facilitate the reduction of the computational workload associated with biometric identification transactions in unprotected biometric systems (i.e.unprotected template indexing), which are prone to unauthorised attacks.The latter has motivated the scientific literature to investigate new customised procedures capable of performing the protected template indexing while reducing the overall computational effort per biometric identification transaction.

C. Biometric template protection
Biometric template protection schemes allow protecting biometric references (i.e.biometric templates) in an unprotected storage environment of a biometric system.Once they are protected, a set of properties are expected to be inherent to the transformed or protected templates constraining the flexibility of the biometric processing pipeline compared to unprotected templates.Comprehensive surveys on this field can be found in [10], [42], [11].Generally, template protection methods are categorised as cancelable biometrics and biometric cryptosystems.The former employ transformations in the signal or feature domain that allow biometric comparison in the transformed (encrypted) domain [43].The latter (e.g.fuzzy vault schemes [13]) usually bind a key to a biometric feature vector resulting in a protected template.Thus, the biometric comparison is then performed indirectly by verifying the correctness of a retrieved key [44].In particular, homomorphic encryption-based template protection schemes are distinguished as biometric cryptosystems whose specific designs allow computing operations directly in the encrypted domain with results comparable to those in the plaintext domain (i.e.unprotected domain) [45].The challenge of unprotected templates being replaced by protected templates leads to requirements or properties which must be fulfilled according to ISO/IEC IS 24745 [42]: Irreversibility: The infeasibility of reconstructing the original biometric sample given a protected template.This type of property guarantees the privacy of the users' data (e.g.avoiding dislocating the subject's ethnic information) and additionally, the security of the system is increased against e.g.presentation attacks and face reconstruction from deep templates.Unlinkability: The infeasibility of determining if two or more protected templates were derived from the same biometric instance, e.g.face.By fulfilling this property, cross-matching across different databases is prevented.Renewability: The possibility of revoking old protected templates and creating new ones from the same biometric instance and/or sample, e.g.face image.With this property fulfilled, it is possible to revoke and re-generate new templates in case the database is compromised.Performance preservation the requirement of the biometric performance not being significantly impaired by the protection scheme.
Tab.I lists the most relevant scientific works on biometric template protection for biometric identification systems based on a single biometric characteristic.The approaches have been analysed in terms of efficient comparison (i.e.workload reduction) and biometric performance.Scientific works on biometric cryptosystems for identification [35], [34], [38] have been commonly focused on providing evidence of practical applicability.The majority of them have contributed to reducing the effort at a one-to-one comparison level by feature transformation while other approaches [36], [39] worked on the reduction of one-to-many comparisons.It is well-known that cancelable schemes appeared to be more suitable in an identification scenario [15], in contrast to biometric cryptosystems (e.g.[41]).That is due to the fact that the design of cancelable biometrics does not require comparison strategies that usually enable the non-flexibility of launching non-arithmetic operations [46] or verifying the correctness of a retrieved key [44].From a practical perspective, cancelable approaches have been therefore successfully considered over identification scenarios for different biometric characteristics (e.g.face, iris, and fingerprint).As mentioned above, these schemes introduce non-invertible transformations at the feature level which usually allow retaining efficient biometric comparators of the corresponding unprotected systems.This way, the majority of published cancelable schemes applied transformations in the feature domain while maintaining acceptable biometric perfor- mance and low computational workload.Over the past years, some feature transformations (e.g.BioHashing [33]) covered discriminative power-based gaps addressing the indexing protected templates with an identification rate at the rank 1 (R-1).Also, the locality sensitive hashing (LSH) [47] nature has recently been exploited and designed to obtain compact noninvertible features (e.g.[14], [32]) where similarly protected templates are more likely to have the same hash collision compared to dissimilar ones.
The described solutions applied workload reduction through an acceleration of a one-to-one comparison.In contrast, other researchers (e.g.[31], [15]) have explored computational workload reduction to decrease the number of one-to-many comparisons which dominates the overall computational effort in biometric identification transactions [37].More precisely, Osorio-Roig et al [15] proposed recently the retrieval of cancelable deep face templates based on their frequent binary patterns.The design of this type of retrieval enabled the use of different cancelable biometric template protection schemes.To sum up, all published works on cancelable biometric template protection for biometric identification worked on an exhaustive search when only feature transformation was employed.Whereas other works reduced the one-to-many search (i.e.pre-selection-based approaches), such schemes are usually not flexible or not designed to work on different biometric characteristics.In addition, some generic multi-biometric indexing methods suitable to work only on unprotected domains have been proposed e.g. in [19], [48], [49].

III. PROPOSED SYSTEM
Consider a biometric enrolment database containing references protected by cancelable schemes1 of N data subjects for m different biometric characteristics or instances.A trivial search process for a single biometric identification transaction would be to conduct the comparisons exhaustively, i.e. the workload of a baseline system is estimated as W baseline = N • m comparisons for all biometric characteristics.In fact, for an improvement of the biometric performance or workload reduction (see [19]), e.g.fusing the scores using one of the traditional strategies (such as score or rank level fusion) mentioned in Sect.II-A, the workload would be dominated by comparisons done exhaustively.As an alternative to the multi-biometric exhaustive search in the protected domain, this work extends the proof-of-concept of frequent binary patterns [15] to indexing multi-biometric cancelable references by employing strategies of biometric information fusion described in the Sect.II-A.In a nutshell, the concept of frequent binary patterns is employed as a multi-biometric efficient binning scheme where each bin (i.e. a single frequent binary pattern) is built by fusing m representations from protected reference templates and allows for indexing them in a single biometric identification transaction.Fig. 1 presents a conceptual overview of the proposed scheme.The design of the multi-biometric binning scheme is template-protectionscheme and biometric characteristic-agnostic which makes it easy to work across different cancelable biometrics extracting ... ... binary representations.Sect.III-A provides details on the approach that computes frequent binary patterns, Sect.III-B describes three strategies of information fusion that result in stable frequent binary patterns for indexing, Sect.III-C describes the retrieval process for each type of information fusion.Sect.III-D discusses the obtained workload reduction.

A. Frequent binary pattern extraction
Frequent binary patterns can be defined in a general concept for the enrolment and retrieval processes, respectively.Formally, the frequent binary patterns can be extracted from a binary representation as follows: let f ∈ {0, 1} n be a bitstring of size n and k < n a given frequent pattern length.A set of unique binary patterns P = {p 1 , . . ., p L }, each of length k can be computed over f by sampling in a sliding window the consecutive k bits starting from positions [0, . . ., n − k] with stride 1.In addition, let O = {o 1 , . . ., o L } be the set of occurrences of each p i ∈ P. Obviously, there is a direct relation between O and P: for each p i ∈ P there exists an o i ∈ O which denotes the number of occurrences of p i in f .Therefore, for a general retrieval process, consider a function FP(•) that extracts the set P ordered descending according to O.

B. Indexing multi-biometric frequent binary patterns
Conceptually, as mentioned above, frequent binary patterns can be extracted only from binary representations.Therefore, deciding which type of information to fuse from the protected references before or after extracting the patterns could impact the efficacy of the proposed binning scheme.Introducing known and simple fusion strategies (e.g.concatenation) on intelligent and convenient steps increases the stability and the discriminative power of the procedure of frequent binary pattern extraction, thereby improving the overall results of the proposed system in terms of biometric performance and computational workload.
Formally, let R i = {r 1 i , . . ., r m i } be the set of data of the subject i ∈ {1, . . ., N } in the enrolment database, where each r j i denotes a protected binary reference associated with the biometric characteristic j ∈ {1, . . ., m}.Given a fixed frequent binary pattern length of k bits, the goal is to build a multibiometric and efficient binning scheme over the base of stable frequent binary patterns successfully extracted on R i .For enrolment, this work considered the fusion strategies at two levels based on the concept of frequent binary patterns: feature and representation level.The former pipeline introduces the concatenation of protected binary references corresponding to different m biometric characteristics.Here, the concatenation acts as doubling the feature dimension by keeping all the elements from the input features.The latter shows the fusion across the maximum binary patterns successfully mapped from individual protected binary references corresponding to m biometric characteristics.

Feature-level Indexing
be the concatenation of m protected binary references of the i-th subject.B i can be then mapped to an individual bin b i which is computed by max(FP(B i )) → b i given a fixed k, as explained in Sect.III-A.That is, the set of data subjects is indexed with at most 2 k bins.Representation-level Indexing each r j i ∈ R i can be independently mapped by the function FP(r j i ) → P j i , resulting in at most 2 k patterns.In this context, two fusion approaches are considered: 1) Ranked-codes: max(P j i ) → b i a single binary pattern resulting in the most ranked frequent binary pattern extracted from the set {P j i } m j=1 is considered as a stable bin for indexing.
2) XOR-codes: the bin b i is constructed from the bitwise XOR operation between the binary patterns with the maximum occurrence in each P j i with 1 ≤ j ≤ m, i.e.XOR(max(P j i )) → b i .

C. Multi-biometric retrieval by fusion strategy
As explained in Sect.III-A, for a general retrieval process, frequent binary patterns are extracted preserving their order of occurrence.It is expected that the pattern with the highest occurrence provides a better chance to find the correct candidate subject than patterns leading with low occurrence as showcased in [15].In a retrieval step, this parameter (i.e.pattern with the highest occurrence) would be estimated on an incremental search for those p patterns with the highest occurrence.For a concrete example, consider 2 3 patterns: P = {p 1 , p 2 , . . ., p 8 }, extracted by the function FP(•) given k = 3.A threshold t with 1 ≤ t ≤ 2 k is determined on P and represents the maximum number of bins that can be visited for a biometric probe.Note that this parameter (t) can easily be controlled by the binning scheme and is independent of the retrieval strategy employed.Also, extracted patterns can only take advantage of their orderings which can be influenced by the retrieval strategy employed (e.g.type of fusion).In this regard, all proposed retrieval strategies employ a scorelevel fusion in a multi-biometric identification transaction once a corresponding bin is determined.In particular, a sum-rule fusion is applied among normalised similarity scores computed from each biometric characteristic.This type of fusion has been utilised in multi-biometric indexing schemes (e.g.[19]) and has also contributed to very good biometric performance in general (see [50] and ISO/IEC TR 24722 [21]).
In this work, three retrieval strategies, one for each type of information fusion are proposed.Firstly, we consider the fact that a binning scheme can be created using one of the strategies described in Sect.III-B.In a retrieval scenario, let Z = {z 1 , . . ., z m } be the set of protected biometric templates for a probe subject, where each z j denotes a binary representation for each of the m biometric characteristics.The key idea is that the proposed retrieval schemes offer different orderings and representations of the extracted frequent binary patterns.Subsequently, a parameter t can be empirically computed in a multi-biometric identification transaction (see Sect.V-B), thereby reducing the system workload while preserving a trade-off between biometric performance, efficiency and privacy.Feature-level Retrieval we follow a similar idea to that of feature-level indexing, as explained above in Sect.III-B.Let B = z 1 ∥. ..∥z m be the concatenation of all z j ∈ Z and a fixed k, the retrieval strategy searches the database for bins belonging to the ordered set P ← FP(B).The final candidate list is therefore composed of the identities associated with the retrieved bins in P. Representation-level Retrieval in contrast to the featurelevel based retrieval, this retrieval pipeline allows searching a t by handling the binary patterns extracted per biometric characteristic.Said patterns are computed as follows: 1) Ranked-codes: the database is searched for the highest ranked binary patterns of each P j ← FP(z j ) and the identities associated with those existing patterns make up the final candidate list.2) XOR-codes: the database is searched for those binary patterns resulting from the bitwise XOR operation among all possible pairs of binary patterns that belong to different P j .Note that the bitwise XOR operations are computed over at most m • 2 k number of pattern pairings that can be constructed from {P j } m j=1 .

D. Computational workload reduction
Our design, which is agnostic with respect to the type of biometric characteristics and cancelable schemes allows searching different biometric characteristics in a single biometric transaction.Therefore, the number of bins visited as well as the number of protected templates stored at each bin is expected to be the same per biometric characteristic.To that end, as mentioned in Sect.III-C, a threshold t may be defined across m types of biometric characteristics used.Although this parameter is easily managed by the multibiometric binning scheme, a computational workload cost may be noticed depending on the biometric characteristics involved (e.g.face and iris, or face and fingerprint), the workload of the individual biometric characteristics, and the strategy of fusion used for retrieval and indexing, respectively.
The computational workload W proposed of an identification transaction (measured in terms of the number of necessary template comparisons) in the proposed scheme, can be expressed as follows: where 1 ≤ t ≤ 2 k denotes a threshold for the maximum number of bins or frequent binary patterns visited in a retrieval step for a fixed k and |b i | the number of protected templates associated with the biometric characteristics m involved.Note that k implicitly is included in the Eq. 1 describing a fixed length in the search for frequent binary patterns (Sect.III-A), and is expected to have an effect on the computational workload along with the biometric performance (see Sect.V).This trend is shown theoretically in Fig. 2. According to Fig. 2, it should be observed that larger k appears to provide a discriminating effect on the built bins, reducing the number of protected templates stored within a bin and thus the overall computational workload.However, some deterioration in biometric performance is observed while maintaining a low workload.Additionally, Eq. 1 shows the relation between the computation of the overall computational workload and the types of biometric characteristics involved in the multi-biometric binning scheme.Fig. 3 theoretically visualizes said relation for e.g. two biometric characteristics involved (BC 1 and BC 2 ) in a range of computed individual workloads.Note that individual workloads can be computed on the proposed scheme using a single biometric characteristic (i.e.uni-modal biometric system).As observed, the overall computational workload appears to be directly proportional to the individual workloads corresponding to each biometric characteristic: W proposed increases with the workloads of the types of biometric characteristics involved.Also, this trend allows some biometric characteristics to take advantage of the unbalanced workloads among individual biometric characteristics, e.g.BC 1 over BC 2 in this case.However, an improvement in overall biometric performance is expected to be achieved, albeit with a slight increase in the overall computational workload.In summary, the key idea behind Eq. 1 is to reduce the computational workload dominated by the cost of comparisons carried out exhaustively.Hence, it is expected that W proposed ≪ W baseline , reducing the penetration rate in the search.An upper bound of W proposed in Eq. 1 is reached,  [55] 204 757 when the system retrieves all bins, resulting in an exhaustive search.In contrast, the best case is when the biometric probe is in the first bin retrieved (i.e.t = 1) and this contains the fewest number of protected multi-biometric templates.

E. Privacy protection
In the proposed system, indexing is performed on protected biometric templates.Obtaining indexes from these cancelable binary templates offers the advantage that the privacy protection of the underlying cancelable scheme is not impaired by the indexing scheme.Recently, it has been shown that indexing methods can leak sensitive information, in particular, if additional indexing data is extracted from unprotected biometric templates [51].In contrast, the proposed scheme extracts the indexing data from the protected templates.This means the privacy protection capability of the used cancelable biometrics scheme is maintained which is a major advantage of the presented indexing method.Precisely, requirements of irreversibility, unlinkability, and renewability are retained from the cancelable biometric scheme.Obviously, the frequent binary patterns extracted from the protected templates do not comprise any additional sensitive information that could be leveraged by an attacker.Due to this reason, the experiments will only focus on biometric performance.For privacy protection analysis the interested reader is referred to the corresponding publication of used cancelable schemes.

IV. EXPERIMENTAL SETUP
This section describes a detailed setup of the experiments conducted on privacy-preserving multi-biometric indexing.Sect IV-A describes the datasets together with the different biometric characteristics employed in this investigation, Sect.IV-B provides details of the extraction process of deep templates, Sect.IV-C details the cancelable template protection schemes, while Sect.IV-E provides the metrics for the evaluation of the proposed system.

A. Databases
For biometric identification experiments where the workload reduction across indexing schemes is analysed, large-scale databases should be considered.Since large-scale databases are not available to researchers, we created a composite database using selected biometric characteristics.This type of database allows operational systems to work independently of the biometric characteristics and their feature representations.A similar concept was utilised in [19].Tab.II shows an  overview of the databases used in terms of the number of instances and samples.Note that we selected the three most common types of biometric characteristics, i.e. face, fingerprint, and iris for our research.Details of the selected biometric characteristics and their databases are described as follows: Face: LFW [52] database is focusing on the large-scale unconstrained face recognition problem.It comprises 13,233 face images captured in the wild from 5,749 subjects collected from the web where 1,680 subjects are represented with two or more images and 4,069 subjects are represented with a single sample.In our experiments, we used only 1,170 identities from the group containing more than one face sample.CR-FIQA(L) [56] as a quality measure has been utilised as a filtering step for selecting the subset of identities with their corresponding samples.Fingerprint: MCYT [53] database containing only fingerprint images captured with an optical capture device is used.This dataset contains all 10 fingers from 330 subjects and 12 samples for each finger, for a total of 39,600 samples.For the experimental protocol, each fingerprint is considered a different biometric instance and is therefore treated as a separate subject.In particular, 1,170 identities are filtered out by using the quality factor NFIQ 2.0 [57].Iris: a mixed iris database was designed to achieve a balanced number of identities with respect to the other biometric characteristics.CASIA-Iris-Thousand [54] database and BioSecure [55] database, both containing images captured in the near-infrared light spectrum, were used.The former contains 1,000 subjects with 2,000 instances, each one represented with 10 samples from each right and left eye.The latter comprises 210 subjects with 420 instances, each one containing 4 samples from each right and left eye.Similar to the fingerprint biometric characteristic, each iris instance was considered a separate identity.In our experiments, a mixed subset was constructed up to a total of 1,170 identities.Iris samples and instances were discarded taking into account different criteria: segmentation errors that led to a bad normalisation step, samples containing glasses, critical images where the visible iris area did not represent more than 70% of the usable iris area, and other quality measures with less critical behaviour such as the iris-sclera, iris-pupil contrast, and the iris-pupil ratio.Note that all quality measures analysed were evaluated and interpreted according to ISO/IEC 29794-6 [58].For the quality assessment of iris samples, an open-source software2 BIQT-Iris was utilised, which reports all the quality measures described in ISO/IEC 29794-6.
Note that 50 identities of 1,170 per database are selected for the score normalisation process and the remaining 1,120 are selected for biometric identification experiments.It is worth noting that biometric identification scenarios are more challenging than verification scenarios as the chance of a false positive can easily increase with the number of comparisons [26].Thus, for the selection of identities per biometric characteristic, the correlation between those biometric samples that produce worse similarity scores (highest chances of false positive in a critical operational point) and the different quality measures were also analysed.It should be noted that quality metrics were evaluated in order to keep only those samples with the best quality.To sum up, it is reasonable that biometric identification systems in real applications may operate with samples that provide acceptable quality in accordance with evaluation standards.Even more when biometric template protection and indexing schemes are employed.For the evaluations of the proposed multi-biometric systems, a database merged with the identities selected independently for each biometric characteristic (face, fingerprint, and iris) is constructed.Fig. 4 shows some example images from the databases selected per biometric characteristic.

B. Deep templates and pre-processing
For the experimental analysis, embeddings extracted by the current state-of-the-art DNN-based recognition systems per biometric characteristic are considered.All embeddings utilised consist of 512 floating-point values.Note that the features extracted per biometric characteristic are balanced in terms of the number of dimensions which allows the indexing scheme to produce the same chances of binary pattern search when they are fused at e.g. the feature level.Details of the extraction process and pre-processing per biometric characteristic are described as follows: Face: ElasticFace [59] represents a state-of-the-art face recognition system.Features are extracted from the pretrained model ElasticFace made available by the authors 3 .Fingerprint: deep fingerprint fixed-length representations are extracted from the open-source software introduced in [60].Note that fingerprint embeddings are extracted by using the training on the texture branch.
Iris: a deep iris representation extractor presented in [61] was used to extract iris embeddings.To that end, the approach proposed by [61] was trained on subsets of the CASIA-Iris-Thousand [54] and BioSecure [55] databases, respectively, from scratch.Note that those instances selected for training were not included in the set of instances for testing that contributed to the biometric identification experiments in this paper.Specifically, for the set of training, 200 instances 4 and 818 instances 5 from BioSecure [55] and CASIA-Iris-Thousand [54], respectively, were selected randomly.ResNet50 [62] architecture is used as the backbone to extract iris feature representations and ArcFace [63] loss function was employed in the training process.Iris images were pre-processed with the traditional approaches: iris segmentation was applied by using the Viterbi [64] algorithm available in the opensource OSIRIS [65], iris textures were normalised according to the rubbersheet model [66], and subsequently, enhanced by applying Contrast Limited Adaptive Histogram Equalization(CLAHE) [67].To sum up, it is important to note that any specific preprocessing like alignment, or type of input to the DNN was considered as described in their corresponding articles of reference.Furthermore, original embeddings extracted per biometric characteristic are converted to 512 binary-values feature vectors (i.e.unprotected baseline system) by using a simple sign function with threshold 0. This type of representation is feasible for the design of the proposed scheme enabling the one-to-one comparison via hamming distance.

C. Cancelable schemes
Biometric template protection approaches representing the current state-of-the-art for cancelable schemes have been used in these experiments.In particular, the so-called BioHashing [68] and a single instance of the Locality Sensitive Hashing [17] based on Index-of-Maximum Hashing with Gaussian Random Projection (IoM-GRP).The former yields output representations containing 512 binary-point values, while the latter comprises 512 integer-point values.In order to facilitate the design agnostic w.r.t. the output of the cancelable scheme (binary representation) prior to the application of the proposed indexing scheme, output representations of the IoM-GRP approach were binarised prior to the frequent binary pattern extraction process.To that end, each integer value is encoded in n bits which are computed on a one-hot encoding by using the maximum number of Gaussian Random Projection vectors (q) for all the IoM-GRP integer representations.Finally, a binary representation with length n•m bits, where m represents the number of Gaussian Random Matrices or length of the integer representation, can be obtained.In these experiments, we used q = 16 and m = 512 to obtain a binary vector of size 2,048 bits.Note that for the computation of the similarity score function for a single biometric identification transaction, 4 those instances containing more than 3 samples 5 those instances containing more than 5 samples this scheme employs its own comparator based on the number of collisions through integers.In particular, BioHashing [68] employs hamming distance.Overall, all protected templates have been used on stolen-token scenarios where non-mated comparisons have access to the genuine users' secret key and use this key with the impostors' own deep features.

D. Proposed system configurations
Biometric identification experiments including the exhaustive search, i.e. baseline workload (W baseline ), and the proposed indexing scheme (i.e. at the feature-and representationlevel) were conducted using 10-fold cross-validation for closed-set and open-set scenarios, respectively.For each fold, two samples per instance are randomly selected, one for enrolment and the other for search.It should be noted that the same samples (same randomness) selected for enrolment and search for each fold are maintained across the configurations of the proposed indexing schemes.Note also that the proposed multi-biometric approach applies to all possible combinations of two types of biometric characteristics and to all three types of biometric characteristics together.Moreover, for the step of score normalisation, the Z-score method is utilised as done in [19], which uses the arithmetic mean and standard deviation of the scores data.

E. Evaluation metrics
The experimental evaluation is conducted according to two key aspects which are considered using methods and metrics standardised from the ISO/IEC 19795-1:2021 [18] and supported by others which are commonly reported in the scientific literature: • Biometric performance: for the closed-set scenario, the hit-rate (H-R), the proportion of subjects for which the corresponding subject identifier is in the subset of candidates retrieved by the proposed indexing scheme; for the open-set scenario, the detection error trade-off (DET) curves between the false negative identification rate (FNIR) and false positive identification rate (FPIR).• Computational workload reduction: average proportion of the total number of references that are retrieved per identification transaction (denoted W ) compared to a baseline workload (i.e. an exhaustive search).It is worth noting that W is theoretically defined in Sect.III-D.

V. RESULTS AND DISCUSSION
In this section, the experimental results are described.Firstly, in Sect.V-A, the proof-of-concept of frequent binary patterns for indexing deep cancelable templates is empirically validated to work on a single-biometric characteristic: face, iris, and fingerprint, against a baseline workload (i.e.exhaustive search).Subsequently, Sect.V-B shows the results of indexing by combining different biometric characteristics at different levels.It is worth noting that all the figures (plots) utilise nomenclatures to refer to the different types of biometric characteristics: FA(face), FP(fingerprint), and IR(iris).Also, different nomenclatures to refer to different statistical

A. Single-biometric characteristic
Tab. III shows the effect of the length of the frequent pattern (k) in relation to the hit rate (H-R) and the system workload (W) empirically computed for a set of identification transactions over closed-set scenario.Note that k has been only shown for the best configuration and for a final value of k (i.e.k=8).An extended overview for all k-combinations can be found in the supplementary material Tab.VII.In the context of the workload computation, two workload reductions representing a lower bound (W l ) and an upper bound (W u ) are estimated.The former considers the lowest number of comparisons equitably distributed among bins without considering their standard deviations, while the latter considers an increased workload taking into account the standard deviations.Note that for a realistic scenario (e.g.open-set scenario), the overall workload would be limited to the upper limit of computational workload (see W u on Tab.III) that can be easily controlled by a fixed number of bins for a biometric identification transaction.In addition to the closedset scenario evaluations, Tab.IV shows open-set results for  Additionally, it can be observed that the workload is inversely proportional to the length of the frequent pattern (k): workload decreases as the length increases, while some H-R values are compromised.A similar trend is theoretically shown in Fig. 2 (Sect.III-D).This observation is to be expected, as bins constructed from longer lengths are more discriminative and can reduce the number of candidates in a comparison step.Therefore, this type of binning design makes the indexing scheme highly dependent on the intra-class and inter-class variance of each biometric characteristic.
Focusing on the open-set results in Tab.IV, at a fixed number of bins (see column #Bins in Tab.IV), it can be observed that indexing schemes for individual biometric characteristics do not achieve similar biometric performances with respect to their corresponding exhaustive searches.However, their workload reductions are remarkable with respect to the baseline workload.Also, the proposed multi-biometric indexing scheme is expected to outperform the biometric performance of the retrieved individual biometric characteristics, while maintaining the overall workload of the system.

B. Multi-biometric characteristics
While Sect.V-A validated the concept of frequent binary patterns as a solution agnostic w.r.t.types of biometric characteristics (i.e.face, iris, and fingerprint) and cancelable biometric template protection schemes (with binary representation), this section shows the evaluation of different fusion strategies proposed in Sect.III.Note that the proposed schemes allow the retrieval of protected multi-biometric templates in a single biometric transaction.Initially, the multi-biometric indexing results in a closed-set scenario are depicted in Tab.V for the best k-combinations across the proposed indexing approaches for the BioHashing scheme.An extended overview of all kcombinations and protection schemes can be found in the supplementary material Tab.VIII (feature-concatenation indexing), Tab.IX (most ranked-code indexing), and Tab.X (XOR-code indexing).Similar to single-biometric characteristic (Sect.V-A), the closed-set scenario evaluation allows estimating a threshold in terms of bins visited (see column #Visited-patterns on the Tab.V in supplementary materials or see column #Bins in Tab.VI) which can then be set in further open-set scenario evaluations.Since the order of the combinations between the biometric characteristics does not affect the final workload of the system in the closed-set scenario, a single combination for two and three biometric characteristics is shown.
As observed in Tab.V, the computational workload required by the proposed multi-biometric techniques increases slightly or greatly depending on the type of strategy and the level at which the data are merged.The proposed multi-biometric approaches based on the highest-ranked code and featureconcatenation indexing improve the overall workload of some independent biometric characteristics, while slightly increasing the individual workload of others.More specifically, this trend is generally observed when designing combinations of types of biometric characteristics centred on the fingerprint.In other words, the Face-Fingerprint combination results in an average workload of approximately W l =∼31%, which represents 16 percentage points lower than the workload of the face as a single biometric characteristic (i.e.W l =∼47% in Tab.III) and approximately 18 percentage points more than the workload yielded individually by the fingerprint (i.e.W l =∼13% in Tab.III).Similar trends can be also observed for the combinations with Iris, e.g.Face-Fingerprint and Face-Iris.The above observations have also been modelled theoretically in Fig. 3 and Sect.III-D.We believe that these gaps or imbalances in terms of overall workloads across the biometric characteristic combinations are due to the fact that singlebiometric characteristics (e.g.face or fingerprint) may exhibit different biometric variances (intra-and inter-class).Note that some variations are nearly inevitable and specific for some biometric characteristics, e.g. for fingerprint, environmental conditions during the sample acquisition process and for iris, distance and angle from the sensor.
Subsequently, the evaluation of the open-set scenario is shown in Tab.VI across the proposed multi-biometric indexing approaches for BioHashing.For convenience, the evaluation of each of the individual biometric characteristics indexing systems is also presented.In these experiments, all possible combinations of types of biometric characteristics and orderings are analysed.The results of the other protection schemes can be found in the supplementary material Tab.XI (Baseline) and Tab.XII (IoM-GRP).
Note that, on the one hand, the overall computational workload (W) of the different multi-biometric approaches proposed is not affected by the order of the biometric characteristics involved in the combination, e.g.Face-Iris or Iris-Face.On the other hand, the workload (W) of the systems generally depends on the type of biometric characteristics used in the combination process, similar to what was observed for the closed-set scenario, e.g.Face-Iris results in a higher W than the Fingerprint-Iris.Furthermore, the proposed multi-biometric schemes outperform single-biometric characteristic indexing pipelines in terms of biometric performance, while producing an approximate average W of the individual BCs.Note the imbalances in terms of W between multi-biometric and singlebiometric characteristic systems.In particular, and depending on the multi-biometric strategy, the FNIR produced by the single-biometric characteristic approaches is reduced down to 19.81% for high-security thresholds (i.e.FPIR = 0.01%).With regard to the above results, we also observe that the best trade-off between W and biometric performance is achieved by combining three biometric characteristics, e.g. the rankedcodes approach results in a FNIR = 21.55% at a FPIR = 0.01%, which is approximately up to 53 percentage points less than the FNIR yielded e.g. for Iris at the same operating point (FNIR = 74.42%).These performance trends are confirmed in Fig. 5: The blue DET curves, representing the multibiometric scheme merging three BCs, significantly outperform the remaining curves associated with the individual biometric characteristics for higher security thresholds.DET curves for other protection schemes can be found in the supplementary material Fig. 6 (Baseline) and Fig. 7 (IoM-GRP).

VI. CONCLUSIONS
A multi-biometric indexing scheme for binning and retrieving protected biometric templates is proposed.We show that the proposed approach is agnostic across biometric characteristics and cancelable biometric schemes.Focusing on unprotected biometric systems, some published works have reported results in terms of workload reduction that go beyond the approach presented [6].Nevertheless, most of these systems are custom-built for specific biometric systems and are not expected to be applicable to other systems.In contrast to these schemes, the proposed system can be used to merge different biometric characteristics, such as face, fingerprint and iris, while protecting the privacy of the subjects.Experimental evaluations compliant with the international metrics defined in the ISO/IEC 19795-1:2021 [18] showed that a protected multibiometric identification system can reduce the computational workload to approximately 57% (indexing up to three types of biometric characteristics) and 53% (indexing up to two types of biometric characteristics), while simultaneously improving the biometric performance at the high-security thresholds of a baseline biometric system.

Fig. 1 .
Fig.1.Conceptual overview of the proposed multi-biometric scheme.Firstly, the system receives different biometric characteristics or instances which are processed by state-of-the-art DNN-based embedding extractors.Subsequently, feature vectors of equal size are protected and encoded in a binary representation by well-established cancelable schemes.Techniques of information fusion are then applied to the protected features along with the concept of frequent binary patterns for the indexing and retrieval steps.Finally, a protected candidate list can be returned taking into account the statistics of their frequent patterns.

Fig. 2 .Fig. 3 .
Fig.2.Effect of k on the trade-off between the biometric performance (γ) and the computational workload (W proposed ).Consider that a biometric performance can be computed in any biometric identification transaction (e.g.γ → hit-rate).

TABLE II SUMMARY
OF THE DATASETS USED IN IDENTIFICATION EXPERIMENTS.

TABLE III CLOSED
-SET SCENARIO OVER SINGLE-BIOMETRIC CHARACTERISTIC.THE BASELINE REPRESENTS THE UNPROTECTED SYSTEM.BOLD VALUES SHOW A TRADE-OFF BETWEEN BIOMETRIC PERFORMANCE AND COMPUTATIONAL WORKLOAD REDUCTION.THE DESCRIPTION OF THE NOMENCLATURE NAMED IN THE COLUMNS IS DETAILED BELOW THE TABLE.
k: Length of the frequent binary pattern, #Comb: Number of possible combinations to be generated given a k, #Comp: Average number of comparisons, Std comp: Standard deviation across the number of comparisons carried out per subject, W l : Lower bound of the computational workload reduction estimated on the average number of comparisons computed per subject, Wu: Upper bound of the computational workload reduction, #Visited-patterns: Average number of binary patterns visited from the probe, Std bins v: Standard deviation across the bins visited per subject, H-R: Hit-Rate.data computed in closed-set scenarios have been employed: #Comp: Average number of comparisons, Std comp: Standard deviation across the comparisons done per subject, #Visitedpatterns: Average number of binary patterns visited from the probe, Std bins v: Standard deviation across the bins visited per subject.

TABLE V CLOSED
SCENARIO RESULTS OF THE PROPOSED MULTI-BIOMETRIC INDEXING SCHEMES ON BIO-HASHING FOR THE BEST PARAMETER K.

TABLE VII CLOSED
-SET SCENARIO OVER SINGLE-BIOMETRIC CHARACTERISTIC.THE BASELINE REPRESENTS THE UNPROTECTED SYSTEM.BOLD VALUES SHOW A TRADE-OFF BETWEEN BIOMETRIC PERFORMANCE AND COMPUTATIONAL WORKLOAD REDUCTION.THE DESCRIPTION OF THE NOMENCLATURE NAMED IN THE COLUMNS IS DETAILED BELOW THE TABLE.Length of the frequent binary pattern, #Comb: Number of possible combinations to be generated given a k, #Comp: Average number of comparisons, Std comp: Standard deviation across the number of comparisons carried out per subject, W l : Lower bound of the computational workload reduction estimated on the average number of comparisons computed per subject, Wu: Upper bound of the computational workload reduction, #Visited-patterns: Average number of binary patterns visited from the probe, Std bins v: Standard deviation across the bins visited per subject, H-R: Hit-Rate.

TABLE VIII CLOSED
SCENARIO OVER MULTI-BIOMETRIC CHARACTERISTICS.RESULTS ARE SHOWN FOR THE FEATURE-CONCATENATION INDEXING.
k: Length of the frequent binary pattern, #Comb: Number of possible combinations to be generated given a k, #Comp: Average number of comparisons, Std comp: Standard deviation across the number of comparisons carried out per subject, W l : Lower bound of the computational workload reduction estimated on the average number of comparisons computed per subject, Wu: Upper bound of the computational workload reduction, #Visited-patterns: Average number of binary patterns visited from the probe, Std bins v: Standard deviation across the bins visited per subject, H-R: Hit-Rate.

TABLE IX CLOSED
SCENARIO OVER MULTI-BIOMETRIC CHARACTERISTICS.RESULTS ARE SHOWN FOR THE MOST RANKED-CODE INDEXING.
k: Length of the frequent binary pattern, #Comb: Number of possible combinations to be generated given a k, #Comp: Average number of comparisons, Std comp: Standard deviation across the number of comparisons carried out per subject, W l : Lower bound of the computational workload reduction estimated on the average number of comparisons computed per subject, Wu: Upper bound of the computational workload reduction, #Visited-patterns: Average number of binary patterns visited from the probe, Std bins v: Standard deviation across the bins visited per subject, H-R: Hit-Rate.

TABLE X CLOSED
SCENARIO OVER MULTI-BIOMETRIC CHARACTERISTICS.RESULTS ARE SHOWN FOR THE XOR-CODE INDEXING.Length of the frequent binary pattern, #Comb: Number of possible combinations to be generated given a k, #Comp: Average number of comparisons, Std comp: Standard deviation across the number of comparisons carried out per subject, W l : Lower bound of the computational workload reduction estimated on the average number of comparisons computed per subject, Wu: Upper bound of the computational workload reduction, #Visited-patterns: Average number of binary patterns visited from the probe, Std bins v: Standard deviation across the bins visited per subject, H-R: Hit-Rate.