Analyzing the Impact of Demographic and Operational Variables on 1-to-Many Face ID Search

Concerns related to the proliferation of automated face recognition technology with the intent of solving or preventing crimes continue to mount. The technology being implicated in wrongful arrest following 1-to-many face identification has heightened scrutiny in the media and spawned lawsuits. We analyze the accuracy of 1-to-many face identification across African-American and Caucasian demographic groups and in the presence of blur and reduced resolution in the probe image, as might occur in “surveillance camera quality” images. Because of the high accuracy of modern face recognition algorithms with “government ID quality” images and the size of available datasets, we use the following indirect metrics to assess the relative propensity for false positive identifications: (1) the traditional d-prime statistic between mated and non-mated score distributions, (2) absolute score difference between thresholds in the high-similarity tail of the non-mated distribution and the low-similarity tail of the mated distribution, and (3) distribution of (mated – non-mated rank-one scores) across the set of probe images. We find that demographic variation patterns in 1-to-many accuracy largely, but not perfectly, follow that observed in 1-to-1 accuracy. We show that, different from the case with 1-to-1 matching accuracy, demographic comparison of 1-to-many accuracy can be affected by different numbers of identities and images across demographics. We also show that increased blur or reduced resolution of the face in the probe image can significantly increase the false positive identification rate. This point is important because the reputation of modern face recognition algorithms for high accuracy in 1-to-many face identification comes from statistics based on analyzing “government ID quality” rather than “surveillance camera quality” images.