Beyond One-Size-Fits-All: Comparing and Selecting Regression Metrics for Robust Model Assessment

In this study, we present a comprehensive overview of common regression metrics, including Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), Coefficient of Determination (R²), and Adjusted R². Each metric serves a unique purpose, from emphasizing large deviations (MSE/RMSE) to quantifying average errors (MAE), relative disparities (MAPE), and explained variance (R² and Adjusted R²). We illustrate the practical significance of these metrics through a case study where three distinct regression models (Linear Regression, Random Forest, and Gradient Boosting) are evaluated. Our findings show that different metrics can yield varying, and sometimes contradictory, assessments of model performance. This demonstrates the importance of choosing metrics that align with domain requirements and error sensitivities. By combining multiple metrics and leveraging both linear and ensemble methods, practitioners can achieve a richer, more accurate understanding of a model’s strengths and limitations, thereby supporting more informed decision-making in real-world regression tasks.