Designing Cross-Race Tests for Forensic Facial Examiners, Super-recognizers, and Face Recognition Algorithms

Humans and machines vary in the accuracy with which they recognize faces of different races. This can impact the fairness of face identification in security and forensic settings. We introduce a protocol for designing a cross-race face identification test for evaluating people (e.g., forensic facial examiners, super-recognizers) and machines with superior face-identification ability. We followed this protocol to create a cross-race test and report the test's benchmarks on untrained human participants and two state-of-the-art face recognition algorithms. The goal of the protocol is to select a relatively small number of challenging test items (facial image comparisons) of two races, with approximately equally challenging items of both races. Item selection consisted of pre-screening with an open-source face recognition algorithm, followed by a second round of prescreening using the performance of untrained human participants. We sampled face-images (Black and White identities) from a large biometric data set and applied the protocol to assemble face comparisons. The protocol yielded a cross-race test with 20 comparison pairs portraying Black and White identities (10 same-identity; 10 different-identity). Untrained participants (54 Black; 51 White) judged whether face-image pairs showed the same or different identities using a 7-point scale. By design, the test proved challenging for untrained participants, with performance comparable across Black and White image pairs for both Black and White participants. Two top-performing face recognition systems from the Face Recognition Vendor Test-ongoing [6] scored perfectly (no errors) on both Black and White face-image pairs from the Cross-Race Test. The human and machine benchmarks established here make this test ideal for evaluating cross-race face recognition bias in people with high levels of skill and training.