Bayesian Rules of Thumb: Robust Uncertainty Quantification in Early Project Cost Estimation

Systems engineers often make use of cost Rules of Thumb in order to estimate cost during early phases of project formulation. These Rules of Thumb typically take the form of a sequence of percentages over which a total cost is allocated across NASA WBS elements. Rules of Thumb can then be used to extrapolate cost from one or more known WBS elements to the remaining unknown WBS elements, assisting early project formulation architecture studies (such as those in JPL's Team X and A Team). A number of issues can arise when generating and using cost Rules of Thumb. For example, many records of project costs consist of incomplete data. Typical methods of dealing with incomplete cost allocation data include (a) ignoring missions with incomplete data, or (b) taking averages of the non-zero percentages across missions, but both of these methods can result in biased estimates if the existence of incomplete data correlates with total mission cost or any particular WBS element. Another common example is cost reported in one or more incorrect WBS elements. This is especially prevalent in smaller missions where it is more common for engineers to perform tasks that fall under the purview of multiple WBS elements. Furthermore, a Rule of Thumb estimate is typically reported as a point estimate; there is no reported uncertainty around the percentages used to generate an allocation. Even in the rare case in which confidence intervals around mean percentages are provided, there may be positive or negative correlations between WBS elements which can skew estimates. Here we attempt to address these problems by formulating probabilistic Rules of Thumb in which a distribution of allocation schemes, rather than a single allocation scheme, is generated. We use a bootstrap imputation method to simultaneously account for uncertainty in the missing data while using all available information contained in the dataset. The imputed datasets are then input into a multivariate Bayesian model which accounts for correlations between WBS elements and properly accounts for uncertainty in the final Rule of Thumb percentages and predictions. We describe the mathematical model and provides snippets of R code utilizing the brms (Bayesian Regression Models using Stan) package. To illustrate this model, we generate a Bayesian Level 2 WBS Cost Rule of Thumb for MIDEX (Medium-Class Explorers) missions with data extracted from NASA's CADRe. We then compare this method's performance with the classical Rule of Thumb method.