How does Rejecting Low-Confidence Decisions in Face Recognition Affect Fairness?

Current face recognition systems are widely used in critical decision-making processes. However, these systems struggle with strong biases and have disparities in the performance of different demographic sub-groups, leading to unfair decisions for the users. On the other hand, decision confidence estimation techniques were introduced recently for biometric recognition systems to enhance the reliability of matching decisions. This work provides the first research to explore the relationship between these two areas by examining how well bias can be mitigated on the decision level through the rejection of low-confidence decisions. In this study, the effect on bias was evaluated with respect to age, gender, and ethnic bias. The experiments were conducted on three current face recognition systems and four state-of-the-art confidence estimation methods. The results demonstrate that fairness can be significantly enhanced by rejecting low-confidence decisions, indicating that uncertain matching decisions have a strong impact on unfair recognition behaviour.