The Use and Misuse of Pre-Trained Generative Large Language Models in Reliability Engineering

Generative Large Language Models (LLMs) have garnered significant attention since the release of ChatGPT in November 2022. Researchers are actively exploring diverse applications to leverage the capabilities of these LLM systems. Within the field of reliability engineering there exists a potential for fruitful utilization of such models. In this paper, we delve into the applications of Large Language Models in reliability engineering, specifically focusing on their impressive language processing capabilities beyond traditional Natural Language Processing (NLP) tasks. Our study aims to evaluate the LLMs' potential in answering complex engineering questions and offering solutions to intricate problems. Additionally, we investigate the limitations of LLMs to understand their boundaries in providing accurate and reliable outputs. The paper emphasizes the significance of prompt engineering to enhance the accuracy and reliability of LLMs for improved performance in quantitative tasks. By incorporating minor prompt engineering techniques, the Large Language Models (LLMs), especially GPT-4, exhibited promising performance in answering Certified Reliability Engineer (CRE) exam questions. Our study involves an analysis of the errors made by the LLMs, allowing for a understanding of their limitations. Drawing from our findings, we provide recommendations on the appropriate application and areas to exercise caution when employing LLMs in the field of reliability engineering. These insights aim to guide practitioners in maximizing the benefits of LLMs while being mindful of their limitations and potential pitfalls. It is important to note that Generative AI and LLMs are rapidly evolving, and the evaluation conducted in this study reflects the test results at the time of writing. We anticipate that LLM responses may vary in the future. We are currently conducting research on developing applications based on LLMs to support the daily tasks of reliability engineers. We are excited about the possibilities and look forward to sharing our outcomes and contributing to the community.