A Novel Number Representation and Its Hardware Support for Accurate Low-Bit Quantization on Large Recommender Systems

Deep learning based recommender systems with large embedding tables have become pivotal for web content recommendation. However, the growing size of those tables, reaching tens of gigabytes or even terabytes, presents a tough challenge for conducting inferences on resource-constrained hardware. In this paper, we present a novel 6-bit fixed-point number representation format for more precise quantization on recommender models. The proposed format is specifically designed to accommodate the nonuniform weight distribution inside those huge embedding tables. To further alleviate the model size, the well-known K-means quantization technique is utilized for 4-bit quantization and beyond. Moreover, we also propose dedicated hardware decoder architectures for both 6-bit and 4-bit quantization to ensure efficient runtime inference. Experimental results show that the proposed low-bit (8~3-bit) quantization techniques on embedding tables yield 4~10.7x model size reduction with minor accuracy loss as compared to the original FP32 model. Therefore, the proposed number representation format and low-bit quantization techniques can effectively and drastically reduce the model size of large recommender systems with a very low area cost while still keeping the accuracy loss minimized.