How Well Can Language Models Understand Politeness?

Politeness plays a key role in social communications. Previous work proposed an SVM-based computational method for predicting politeness using linguistic features on a corpus that contains Wikipedia and Stack Exchange requests data. To extend this prior work, we focus on evaluating the performance of state-of-the-art language models on politeness prediction using the same dataset. Two models are applied in this study. First, we fine-tune BERT on politeness data and then use the fine-tuned model for politeness prediction. Second, we use ChatGPT to predict politeness. The results show that both fine-tuned BERT and ChatGPT achieved better results than the state-of-the-art results on both Wikipedia and Stack Exchange data. Fine-tuned BERT outperforms zero shot ChatGPT, but ChatGPT can provide explanations for its prediction. Moreover, fine-tuned BERT outperforms human-level performance by 2.28% on Wikipedia corpus.