VarVE: Bringing SIMD Performance to Variable-Width Values

Processor datapaths grew from 4 to 512 bits via Single-Instruction-Multiple-Data (SIMD) parallelism. SIMD applies the same operation to multiple values, which increases performance and reduces the instruction count. However, these evolvements do not provide support for variable-width values, so programmers ‘pad’ values to align to outliers, wasting the upper bits with zeros in both registers and datapath. We propose VarVE, a vector instruction set extension built upon the state-of-the-art vector-length agnostic SIMD instruction set: ARM SVE. VarVE provides native support for variable-width values within a vector, avoiding padding waste, thus making better use of the SIMD datapath. VarVE's design enables a flexible strip mining model with a variety of optimizations. Evaluation of VarVE shows 60x speedup over ARM in kernels with element packing and unpacking, and 1.3x - 5.4x speedup over SVE for pure-compute filtering in TPC-H benchmarks. VarVE also achieves 2x speedup on a neural network inference task. All these results exemplify VarVE's general ability to improve datapath and memory system efficiency.