Efficient Design Space Exploration for the BOOM Using SAC-Based Reinforcement Learning

Design space exploration (DSE) is crucial for optimizing the performance, power, and area (PPA) of CPU microarchitectures ( $\mu $ -archs). While various machine learning (ML) algorithms have been applied to the $\mu $ -arch DSE problem, the potential of reinforcement learning (RL) remains underexplored. In this article, we propose a novel RL-based approach to address the reduced instruction set computer V (RISC-V) CPU $\mu $ -arch DSE problem. This approach enables dynamic selection and optimization of $\mu $ -arch parameters without relying on predefined modification sequences, thus significantly enhancing exploration flexibility. To address the challenges posed by high-dimensional action spaces and sparse rewards, we use a discrete soft actor-critic (SAC) framework with entropy maximization to promote efficient exploration. In addition, we integrate multistep temporal-difference (TD) learning, an experience replay (ER) buffer, and return normalization to improve sample efficiency and learning stability during training. Our method further aligns optimization with user-defined preferences by normalizing PPA metrics relative to baseline designs. Experimental results on the Berkeley out-of-order machine (BOOM) demonstrate that the proposed approach achieves superior performance compared with state-of-the-art methods, showcasing its effectiveness and efficiency for $\mu $ -arch DSE. Our code is available at https://github.com/exhaust-create/SAC-DSE.