HOLES: Boosting Large Language Models Efficiency with Hardware-Friendly Lossless Encoding

Transformer-based large language models (LLMs) have demonstrated remarkable success; however, their increasing model size poses a challenge due to the widening gap between model size and hardware capacity. To address this, model com-pression techniques have been proposed, but existing compression methods struggle to effectively handle the significant parameter variations (activations and weights) within the model. More-over, current outlier-aware encoding solutions for compression introduce complex logic, leading to limited compression benefits and hardware efficiency. In this context, we present HOLES, a novel algorithm/architecture co-designed solution that utilizes variable-length data representation and metadata for local pa-rameter value processing, offering low hardware overhead and high-performance gains. Our key insight is that only a tiny fraction of activations are outliers demanding high precision representation. This observation enables us to exploit the sparsity of high significant bits within data representation, allowing us to employ an identifier bit to assign the appropriate encoding length, thereby eliminating redundant bit-length footprints. This reduction in data representation based on data characteristics enables a serialized structured data encoding scheme that seam-lessly integrates with existing hardware accelerators, such as systolic arrays. We evaluate HOLES-based accelerators against some existing encoding-based accelerator, and our results demon-strate significant improvements. The HOLES-based accelerator achieves up to 3.98 × speedup and 70.0% energy reduction, while maintaining superior model accuracy.