Leveraging LLMs Towards Assistant-based Support for Industrial Threat Models

Threat models contribute to strengthen the security of an enterprise system by listing the cybersecurity threats that might affect it as well as possible mitigations to such threats. Given the importance of these models, different tools have been devised to support creating or updating threat models. However, these tools focus mainly on the needs of cybersecurity experts, often overlooking non-expert users.To address this gap, in the first part of this paper we present a chat-based LLM assistant to answer both expert and non-expert queries on threat models. The assistant is based on a microservice architecture and uses Retrieval-Augmented Generation (RAG) to extract relevant information from an industrial threat model.Guaranteeing the reliability of an LLM’s answers in sensitive fields such as cybersecurity is paramount. For this reason, in the second part of this paper, we evaluate the LLM assistant using (i) Bleu and Rouge score metrics, (ii) human evaluation, and (iii) an automatic LLM-as-a-judge approach.The evaluation, conducted on 275 test cases, confirms the accuracy of the answers provided by our LLM-based threat model assistant. Additionally, the results of the human and LLM-as-a-judge evaluations are consistent. This corroborates the effectiveness of automatic LLM-as-a-judge methods in assessing the performance of LLM-based industrial solutions.