Applying system readiness levels to cost estimates — A case study

Accurately estimating cost, schedules and expected technical performance of large complex systems and systems of systems (SoS) pre-development is difficult. As programs mature and begin execution, they often plague of cost growth and schedule delays due to unexpected deviations from initial assumptions or complexity. For example, Government Accountability Office's (GAOs) 2016 review of selected weapons systems found that twelve of the 43 programs in the report had cost growth from 4% to 45% as well as schedule delays averaging thirty months. Similarly, the James Webb Space Telescope (JWST) has had multiple over runs from inception.[1,2] As the program continued cost growth and schedule delays due in part to technical maturity and complexity were apparent resulting in a 2011 re-baseline. The launch, now planned for 2018, has jeopardized the science community's ability to collect timely and relevant deep space scientific data requiring the Hubble Space Telescope (HST) to perform significantly beyond its already extended design life. Using System Readiness Levels (SRLs), an emerging technique, in concert with heuristic cost and schedule analysis may provide enhanced capabilities for pre-development cost estimates and schedule completions within the acquisition lifecycle. Up to now, theoretical and analytical approaches are in the literature. This paper investigates actual program cost and schedule history through program development, then comparing it to SRL forecasting techniques to demonstrate its validity. The resulting approach will support future estimating accuracy through higher fidelity information for early decision-making. We research and collect actual program development cost, schedule and technical history for Major Defense Acquisition Programs (MDAPS) and NASA flagship programs; then compare the actual growth metrics to the SRL forecasts. This empirical cross-referencing applied to the SRL framework can support more realistic cost and schedule forecasting early in program development. Evaluation of the data to support a null hypothesis demonstrating a statistically significant approach can provide significant cost and schedule estimating process improvement; with a goal of implementing a validated process early into forecast models. The method can isolate specific areas for Technology Maturity and Risk Reduction (TMRR) focus and actions for specific sub-systems within the system or SoS pre-milestone B for DoD and Phase B for NASA. Moreover, cost and schedule realism, and credibility at the start of “Program of Record” with high confidence realized. The data analyses will provide demonstrated confidence using these methods. A follow up example to demonstrate the method will be shown. Future research will include the addition of more program data points to enhance the robustness of the method, and model enhancements to define cost and schedule drivers within the systems and SoSs.