Is Vanilla Bayesian Optimization Enough for High-Dimensional Architecture Design Optimization?

In the tide of explosive development in artificial intelligence (AI), the design of AI System-on-Chips (SoCs) is an urgently pressing issue that needs to be addressed. The application of Design Space Exploration (DSE) methods is paramount in pursuing a sound microarchitecture design and improving the quality of results. However, the high-dimensional design parameters and huge design space, which normally occur in the complicated SoCs for Large Language Model (LLM) tasks, pose a great challenge to existing techniques. In this paper, a novel and explainable Bayesian optimization-based framework MCT-Explorer is proposed. A Monte Carlo Tree Search (MCTS)-based method is utilized to analyze the importance of design parameters, guide the sampling directions, mitigate low-quality performance modeling issues, and further improve optimization efficiency. Besides, an information-guided multi-objective optimization function is adopted to balance the multiple metrics (e.g., Cycle. Area, and Power) for SoC design. Our approach can provide guiding opinions and deeper insights for parameter optimization, thus transcending previous arts and achieving an explainable model. Experiment results demonstrate the extraordinary performance of our framework in various high-dimensional (up to hundreds of parameters) and complicated LLM SoC designs.