Efficient Inference of Transformers on Bare-Metal Devices with RISC-V Vector Processors

Most neural network frameworks have only limited supports for bare-metal devices, where specific primitives need considering and hand optimizations for such devices incur a substantial development cost. This paper presents an efficient workflow using PyTorch Transformer, where a RISC-V device is used as an example. Models from different frameworks can be efficiently deployed and optimized, where bottlenecks are segmented and optimized using intricsic instructions. Moreever, TVM BYOC (Bring Your Own Codegen) has been adopted for custom optimizations.