Cordial: Cross-row Failure Prediction Method Based on Bank-level Error Locality for HBMs

High Bandwidth Memory (HBM) is a promising solution for overcoming memory bottlenecks in high-performance computing, but it remains susceptible to memory errors. Our empirical study on high-performance computing platforms that serve large AI model training with over 80,000 HBMs, reveals that HBM errors have a high burst rate. As a result, conventional failure prediction methods that heavily rely on historical error data are ineffective. Through investigating the locality of uncorrectable errors (UCEs) of HBMs, we found that failure patterns at the bank level primarily exhibit aggregation tendencies, suggesting that errors in neighboring rows are often related. Based on these insights, we propose Cordial, a cross-row failure prediction method based on bank-level error locality. Cordial adopts a hierarchical approach: it first utilizes bank-level error information to predict the bank-level failure pattern. Our approach classifies bank-level failure patterns into three categories: double-row clustering, single-row clustering (these two are aggregation patterns), and scattered patterns. Then, it leverages spatial locality to guide cross-row predictions for aggregation patterns. Evaluation results show our method improves the F1-score by up to 90.7% and enhances the isolation coverage rate by 47.1%, demonstrating its practical applicability.