A simulation-based comparison of empirical modeling techniques for software metric models of development effort

Empirical models for the management of software development projects have until recently been based, with only limited exceptions, on linear least-squares regression. The continued failure of the resulting empirical models to provide adequate assistance to managers has led to the examination (and even some adoption) of more sophisticated modeling techniques. These techniques have included robust statistical procedures, various forms of neural network models, fuzzy logic, case-based reasoning, and regression trees. This paper describes a simulation-based study on the performance of some of these empirical modeling techniques using a size and effort software metric data set. The models are assessed using a variety of "goodness of fit" measures-assessing the predictive performance on hold-out samples across 50 simulations using both sampling with replacement and without replacement. The relative performances of each technique can be used to select that which is "best" given the desired predictive accuracy criterion. Overall the best performing technique appears to be M-estimation. This suggests that robustness to outliers, in this case at least, may be more important than modeling non-linearities or interactions.