Research on AI Chip Large Model Inference Testing Technology

At present, the performance of AI inference cards in the industry mainly depends on the specification parameters of the chip, and in practice, the performance of inference cards is also related to the inference optimization technology of the chip. In response to this issue, we conducted researches on the specification information of current mainstream chip manufacturers. On the basis of studying existing evaluation indicators, a performance measurement scheme was proposed by using latency and throughput of input output context of as performance indicators, and five products were tested. The experimental results indicate that this scheme can scientifically and reasonably quantitatively evaluate the performance and scalability of AI inference cards.