Software Engineering Institute

Extensive cost overruns in major defense programs are common and studies have implicated poor cost estimation as a main contributor. Research and experience have identified several factors associated with poor cost estimates. These include

• optimistic expectations about achievable program scope and technology that can be delivered on schedule and within budget

• the enormous amount of unknowns and uncertainty that exist at the time when these estimates are made about large-scale, unprecedented systems that take years to develop and deploy

• the heavy reliance, of necessity, on expert judgment

• the difficulty in learning from the experience of previous programs

Many assumptions about the desired end product are made in calculating the estimates. However, the estimation process does not capture information about program change factors that can dramatically influence cost over the lifecycle of program research, development, production, deployment, and sustainment.

In this presentation we describe a new, integrative approach for pre-Milestone A cost estimation, which we call QUELCE (Quantifying Uncertainty in Early Lifecycle Cost Estimation). QUELCE synthesizes scenario building, Bayesian belief network (BBN) modeling and Monte Carlo simulation into an estimation method that quantifies uncertainties, allows subjective inputs, visually depicts influential relationships among change drivers and outputs, and assists with the explicit description and documentation underlying an estimate. We use scenario analysis and dependency structure matrix (DSM) techniques to limit the combinatorial effects of multiple interacting program change drivers to make modeling and analysis more tractable. Representing scenarios as BBNs enables sensitivity analysis, exploration of alternatives, and quantification of uncertainty. The BBNs and Monte Carlo simulation are then used to predict variability of what become the inputs to existing, commercially available methods and tools in use by cost estimators. As a result, interim and final cost estimates are shown as distributions. This, in turn, allows an estimator to associate a probability with any given range within the distribution.

QUELCE aims to provide credible and accurate program cost estimates based on available information about the magnitude of uncertainty and the factors influencing program execution. By making visible the potential changes that may occur during program execution, our approach also supports quick revision of program estimates to mitigate risk and a quicker response to the changes that often arise during a program's lifecycle. The same flexibility enables early consideration of the likely impact of possible future scenarios on the estimates. Intuitive visual representations of the data explicitly model influential relationships and interdependencies among the drivers on which the ultimate estimates depend. Assumptions and constraints underlying the estimates that may not otherwise have been considered are well documented, which contributes to better management of cost, schedule, and adjustments to program scope as more is learned and conditions change. Documenting the basis of an estimate also facilitates updating the estimate during program execution and helps others make informed judgments about estimation accuracy. These features address the challenges of early life cycle cost estimation and make QUELCE a promising approach for effectively producing cost estimates in circumstances with a high degree of uncertainty.

About the Speaker(s)

Jim McCurley is a Senior Member of the Technical Staff at the Software Engineering Institute (SEI). During his 15 years at the SEI, his areas of expertise have included data analysis, statistical modeling, and empirical research methods. For the last several years, he has worked with various DoD agencies involved with the acquisition of large scale systems. From 1999-2005, Jim also worked as a member of the Technical Analysis Team for the CERT Analysis Center.

Robert Stoddard is a Senior Member of the Technical Staff at the Software Engineering Institute (SEI). Robert earned a BS in Business, an MS in Systems Management and is a certified Motorola Six Sigma Master Black Belt. He delivers measurement courses in public and client offerings and provides measurement consulting to external clients.