On the Variability of Statistical Models

The axiom that a good model strikes a good balance between bias and variance may be regarded as a corollary of the principle of parsimony or Occam's Razor. In this context, the bias is measured in training cost, while the variance of, say, a regression model is measured by the cost associated with a validation set. If reducing bias is the goal, one will strive to fetch as complex a model as necessary. Still, complexity is invariably coupled with an increase in variance: greater complexity implies greater variance. In practice, driving training cost to near zero does not pose a fundamental problem; in fact, a sufficiently complex decision tree can drive training cost to zero; however, the problem is controlling the model's variance. We investigate various regression model frameworks, including generalized linear models, Cox proportional hazard models, ARMA, and illustrate how misspecifying a model with ‘excessive’ complexity affects the variance.