Exploring the Capabilities of Large Language Models for the Generation of Safety Cases: the Case of GPT-4

The emergence of large language models (LLMs) and conversational interfaces, exemplified by ChatGPT, is nothing short of revolutionary. While their potential is undeniable across various domains, in this paper we investigate their uncharted territory-the exploration of generating safety cases. Our primary objective is to delve into the existing knowledge base of ChatGPT (specifically, GPT-4), focusing specifically on its understanding of the Goal Structuring Notation (GSN), a well-established notation allowing to visually represent safety cases. Subsequently, in our preliminary evaluation, we perform four distinct experiments with GPT-4. These experiments are designed to assess its capacity for generating safety cases within a defined system and application domain. To measure the performance of GPT-4 in this context, we compare the results it generates with ground-truth safety cases respectively created for an x-ray system and a Machine Learning (ML)-enabled component for tire noise recognition (TNR) component in a vehicle. Our findings indicate that GPT-4 demonstrates the capacity to automatically produce safety arguments that are moderately accurate and reasonable. Furthermore, it also exhibits the capability to automatically generate safety cases whose semantic closely aligns with the one of the reference safety cases used as ground-truths in our experiments.