Investigating Social Biases in Multimodal LLMs

With the rapid advancement of Multimodal Large Language Models (MLLMs) and their ability to integrate multimodal inputs, these models are increasingly being applied to real-world tasks. However, alongside their impressive capabilities, MLLMs often exhibit undesirable characteristics, such as social biases. In this study, we conduct a comprehensive evaluation of bias in MLLMs concerning gender, race, and age attributes. To achieve this, we design a set of visual-question-answering (VQA)-based queries that prompt the models to perform attribute estimation given a face image. We assess these models using class-wise accuracies and bias-related metrics, revealing that while gender biases are relatively minimal, significant biases persist in race and age estimations. Our findings highlight the need for further research to mitigate these biases before deploying MLLMs in real-world applications.