AtRec: Accelerating Recommendation Model Training on CPUs

The popularity of recommendation models and the enhanced AI processing capability of CPUs have provided massive performance opportunities to deliver satisfactory experiences to a large number of users. Unfortunately, existing recommendation model training methods fail to achieve high efficiency due to unique challenges such as dynamic shape and high parallelism. To address the above limitations, we comprehensively study the distinctive characteristics of recommendation models and discover several unexploited optimization opportunities. To exploit such opportunities, we propose
AtRec
, a high-performant recommendation model training engine that significantly accelerates the training process on CPUs. Specifically,
AtRec
presents comprehensive approach of training that employs operator-level and graph-level joint optimizations and runtime optimization. At the operator-level,
AtRec
identifies and optimizes the time-consuming operators, which enables further efficient graph-level optimizations. At the graph-level,
AtRec
conducts an in-depth analysis of the inefficiencies in several frequently used subgraphs, enables further performance improvement via eliminating redundant computations and memory accesses. In addition, to achieve better runtime performance,
AtRec
also identifies inefficiencies prevalent in the current scheduling and proposes runtime batching. The experiment results demonstrate that
AtRec
can significantly outperform state-of-the-art recommendation model training engines. We have open sourced the implementation and corresponding data of
AtRec
to boost research in this direction.