Assessing the Ethical Alignment of Large Language Models Using Deep Embedding Models

Prompting strategies which request justifications behind decisions have been utilized to further explore the ethical alignment of large language models with humans. Recent efforts have assessed the variability in justifications produced for ethically unambiguous scenarios for various prompting styles by employing latent semantic analysis to encode justifications. This manuscript proposes and validates an alternative workflow for this assessment using deep embedding models for encoding. We employ the Massive Text Embedding Benchmark (MPNet) model to embed the justifications produced by GPT-3 for 4 prompting variations. Our results demonstrate that MPNet produces more semantically rich encodings of the produced justifications versus Latent Semantic Analysis. This result is consistent with observations in the literature across similar natural language processing workflows.