Mixed-Precision Neural Networks on RISC-V Cores: ISA Extensions for Multi-Pumped Soft SIMD Operations

Recent advancements in quantization and mixed-precision approaches offers substantial opportunities to improve the speed and energy efficiency of Neural Networks (NN). Research has shown that individual parameters with varying low precision, can attain accuracies comparable to full-precision counterparts. However, modern embedded microprocessors provide very limited support for mixed-precision NNs regarding both Instruction Set Architecture (ISA) extensions and their hardware design for efficient execution of mixed-precision operations, i.e., introducing several performance bottlenecks due to numerous instructions for data packing and unpacking, arithmetic unit under-utilizations etc. In this work, we bring together, for the first time, ISA extensions tailored to mixed-precision hardware optimizations, targeting energy-efficient DNN inference on leading RISC-V CPU architectures. We introduce a hardware-software co-design framework that supports cooperative hardware design, mixed-precision quantization, ISA extensions, and cycle-accurate emulations. At the hardware level, we expand the ALU unit in our micro-architecture for configurable mixed-precision arithmetic operations and implement multi-pumping to reduce execution latency, with soft SIMD optimization for 2-bit operations. At the ISA level, we encode three distinct MAC instructions extending the RISC-V ISA, each for different mixed-precision modes, and expose them to the compiler. Our extensive experimental evaluation over widely used DNNs and datasets, such as CIFAR10 and ImageNet, demonstrates that our framework can achieve, on average, $15 \times$ energy reduction for less than 1% accuracy loss and outperforms the ISA-agnostic state-of-the-art RISC-V cores.