Effects of Language- and Culture-Specific Prompting on ChatGPT

Advanced LLM-based chatbots like ChatGPT are immensely popular, engaging users from diverse cultural backgrounds. Previous studies indicate that when trained with a lot of English data, these chatbots predominantly reflect the nuances of English-speaking cultures, particularly in the US American context. Such a cultural bias may hinder effective communication with users and marginalize non-English cultures. In this paper, we investigate the cultural relevance of chatbots built on the LLMs GPT-3.5-turbo and GPT-4, specifically when used in languages other than English. Our analysis encompassed the five language areas English, German, Spanish, French, and Portuguese and the corresponding ten subcultures English-speaking Great Britain and the USA, German-speaking Germany and Austria, Spanish-speaking Spain and Mexico, French-speaking Canada and France, and Portuguese-speaking Brazil and Portugal. Our assessment of cultural appropriateness employed the ten dimensions from Inglehart and Welzel’s cultural mapping framework. We benchmarked the chatbots’ performances, elicited through specific prompts, against data from the World Value Survey. Subsequently, we developed a unique cultural map according to Inglehart and Welzel’s model. Our findings indicate that GPT-3.5-turbo generally surpassed GPT-4 in cultural alignment, particularly within the German linguistic context. Conversely, the Spanish and Portuguese linguistic regions showed lower cultural alignment. At the subcultural level, German-speaking Germany demonstrated the highest cultural alignment, whereas Spanish-speaking Mexico and Portuguese-speaking Brazil demonstrated the least alignment.