Towards a Verification and Validation Framework for COVID-19 Forecast Models

We present a model verification and validation (V&V) framework to evaluate COVID-19 forecasting models on their report of eight V&V-related components: (1) Conceptual Model, (2) Code and Calculation Verification, (3) Data Validation, (4) Parameter Estimation, (5) Initialization, (6) Uncertainty Estimation, (7) Output Validation, and (8) Model-to-Model Comparison. The framework provides a structured method to evaluate these models based on their reported V&V practices qualitatively. We applied this framework as a checklist for nine models included in the COVID-19 Forecast Hub. One model got the highest score by supporting seven components, while the lowest-ranked model got only two. This framework can serve as part of a larger framework to qualitatively and quantitatively examine COVID-19 models' V&V reported practices and provide credibility for those models that not only perform well but also robust in model construction.