A Quantitative Metric Selection Approach for Time-series Forecasting Foundation Models

The recent emergence of time-series forecasting foundation models allows for prediction of any time series without the need of extra training. To select the best foundation model, researchers often use a random metric (e.g., MAE) and apply it to historical observations whose future values are already known, and subsequently select the model with the lowest errors. A research question naturally arises: which metric is the best to use for choosing a foundation model? This paper proposes a quantitative approach to address this problem. We define and analyze six novel attributes of time-series forecasting metrics, including value emphasis and outlier emphasis, among others. Based on these proposed attributes, we enable analysts to specify their preferences and interests in any attributes, which helps them select the best metric for the downstream foundation model selection task. We demonstrate the usefulness of our approach with two real-world datasets, Air Quality and Electricity.