Mitigating Racial Bias in Facial Expression Recognition Using Enhanced Data Diversification Techniques

With countless new methods and further advancevision, applying the technology to the everyday lives of people has become inevitable. One such use of computer vision models is recognizing facial patterns and expressions. However, as recent studies have indicated, many models interpreting facial expressions have developed biases towards certain races. This is likely a result of skewed training data; the model may have been trained with more images of a certain race than others. Overall, this can cause inaccuracy when the model is presented with people from different backgrounds. To promote a diverse group of data that the model can process, I present a way to balance biases. First, I fine-tuned a hybrid facial recognition model of state-of-the-art convolutional neural network (CNN) models called DeepFace on the emotion recognition dataset FER2013 to recognize the emotions on people’s faces. Next, using FairFace, which is modeled after the large-scale CNN ResNet-34, I extrapolated the races and facial structures for each type of individual, which I then used to determine the bias and skew within the dataset. I did this by comparing the model’s accuracy when applied to different races. Then, I used a user-instructed stable diffusion model called InstructPix2Pix to modify the images and create a more well-represented and diverse dataset. With this method, I was able to balance the dataset effectively and reduce bias in the image recognition model. However, due to issues with the FER2013 dataset images, many of the transformed images were disfigured, meaning the emotion data within the images was lost. This caused the overall accuracy of the model to drop significantly. Future research can evaluate this method using InstructPix 2Pix on different computer vision tasks or with different datasets.