LLM-aided Test Generation for Custom Neural Network Hardware Accelerators

Large Language Models (LLMs) are increasingly being utilized to automate tasks in software development, including test generation. This paper introduces a novel workflow for the automated generation of tests for custom hardware accelerators for Neural Networks (NN) using LLMs. The workflow is designed to handle the unique challenges posed by custom or proprietary accelerators, where the LLM has no prior knowledge about the hardware, and there are not enough code examples to fine-tune the LLM for it. A five-part prompt is proposed to encompass all the necessary information for the LLM to generate accurate tests. The Gemmini accelerator is selected as a use case, and tests are defined and generated using six different LLMs, each one evaluated at 4, 16 and 32-bit floating point precision. For each LLM and precision combination, latency and energy consumption are recorded for each generated test. We evaluate the correctness of the generated tests and the impact of adding the human-in-the-loop to apply minimal corrections to them if necessary. We define an efficiency metric to evaluate the tradeoff between consumed energy and correctly generated tests. The workflow evaluation reveals that LLMs are able to generate tests for custom hardware accelerators, with Codestral and Gemma 2 being the most effective LLMs for these tasks, capable of generating all the proposed tests with minimal human corrections. This workflow offers developers a new and efficient method to accelerate the test generation of custom hardware accelerators for NN, with potential applications in verifying new accelerators or new features of existing ones.