FairFix: Enhancing Fairness of Pre-Trained Deep Neural Networks with Scarce Data Resources

In recent years, the financial technology (Fintech) sector has increasingly recognized the issue of biases in AI models, particularly in areas like credit scoring and fraud detection where computer vision plays a key role. Despite this, eliminating biases in pre-trained vision models is not an easy task. For instance, fine-tuning using unbiased training data is often adopted to eliminate the bias among various racial subgroups in a pre-trained model. It turns out that training samples from different racial subgroups (in practice between the groups with high prediction accuracy and those with low accuracy) may generate opposite gradients on the model leading to the low effectiveness of fine-tuning. This work presents FairFix, a novel method for eliminating bias in unfair Deep Neural Networks models. Specifically, FairFix selectively forgets unfair features learned from the sample of the dominant group and recovers the model with fair data, improving the accuracy of the sensitive group without sacrificing too much accuracy for the dominant group. To evaluate the effectiveness of FairFix, we conducted extensive experiments on various datasets. Our experiment results demonstrate that FairFix outperforms current state-of-the-art fairness improvement methods in terms of both fairness metrics and overall accuracy. This suggests that FairFix is a promising method for improving the fairness of computer vision models.