Re-Pen: Reinforcement Learning-Enforced Penetration Testing for SoC Security Verification

Due to the increasingly complex interaction between the tightly integrated components, reuse of various untrustworthy third-party IPs (3PIPs), and security-unaware design practices, there have been a rising number of reports of system-on-chip (SoC) hardware (HW) vulnerabilities that compromise the security of critical assets. SoC security verification, therefore, is an indispensable part of the verification effort. The existing hardware verification methodologies either presuppose white-box knowledge or scale poorly with increasing design complexity. Hardware penetration testing (pentest) is an emerging gray-box security verification methodology at the register-transfer level (RTL) that is applicable across a wide variety of threat models and addresses many shortcomings of the existing methodologies. In this work, we propose Re-Pen, a novel hardware pentest framework that requires minimal gray-box information from the design specification to achieve significantly better security vulnerability (SV) detection performance than state-of-the-art pentest techniques. At the core of this framework lies a mutation engine that combines the strengths of reinforcement learning (RL) and binary particle swarm optimization (BPSO) in its test pattern mutation strategy to generate intelligent test patterns without manual supervision. This framework significantly reduces the requirement for detailed, manual, expertise-driven adaptations specific to the SoC under test. Through extensive experiments conducted on multiple SoCs, we demonstrate that Re-Pen can reduce vulnerability detection time by up to $3\times $ and achieve a markedly improved consistency compared with the state of the art. Furthermore, Re-Pen was able to detect native security bugs in an open-source SoC. It successfully identified a scenario where, despite a functionally correct hardware implementation, a mistake in the architectural specification allowed privilege escalation from the software layer.