LLM-based Cyber Security Testing for Consumer Internet of Things

With the increasing prevalence of handheld devices in daily life, routers have become essential for connecting devices to the Internet while also becoming prime targets for hacker attacks. Currently, the cybersecurity testing of these devices primarily relies on manual processes. Testing engineers are required to thoroughly understand various regulations, such as EN 303 645 and TS 103 701, and to develop specific testing procedures for different devices. This paper proposes a semi-automated AI system for testing Internet of Things (IoT) devices. The system uses standardized procedures and AI-driven regulatory interpretation to rapidly generate test reports, significantly reducing engineers' time searching for relevant regulations and improving testing efficiency and accuracy. This study evaluates multiple open-source embedding models and the Llama 3.1 70B large language model to develop an Advanced Retrieval-Augmented Generation (Advance-RAG) framework. This framework automatically retrieves test items from cybersecurity regulations and generates preliminary assessments based on self-declarations provided by device manufacturers. It quickly determines whether a device's functionalities comply with specified testing standards. Testing engineers can then upload supporting evidence through the report generation system, providing a comprehensive test report.