Posted on by Measurement & Analysisin
As part of our research related to early acquisition lifecycle cost estimation for the Department of Defense (DoD), my colleagues in the SEI's Software Engineering Measurement & Analysis initiative and I began envisioning a potential solution that would rely heavily on expert judgment of future possible program execution scenarios. Previous to our work on cost estimation, many parametric cost models required domain expert input, but, in our opinion, they did not address alternative scenarios of execution that might occur from Milestone A onward.
Our approach, known as Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE), asks domain experts to provide judgment not only on uncertain cost factors for a nominal program execution scenario, but also for the drivers of cost factors across a set of anticipated scenarios. This blog post describes our efforts to improve the accuracy and reliability of expert judgment within this expanded role of early lifecycle cost estimation.
Our work in cost estimation began two years ago, building upon a review of existing cost estimation and expert judgment research. As an example, we identified an industry consultant, Douglas Hubbard, whose book, How to Measure Anything, presents an approach known as "calibrating your judgment" (my colleague, Dave Zubrow, describes Hubbard's technique in a recent blog post). Hubbard's focus on calibrating expert judgment using "trivial pursuit" exercises led to our team's decision to pursue research into the use of domain-specific reference points to further improve the accuracy and reliability of expert judgment within cost estimation.
Our research on early lifecycle cost estimation for the DoD consists of two tasks:
We hypothesized that a domain-specific approach to calibration training and development of reference points would be necessary to reduce unwanted variation in judgments rendered by experts participating in the QUELCE method. We decided to take a two-pronged approach to improving expert opinion. The first part of the approach involved data mining of DoD program execution experience. The second part of the approach interviewed DoD experts about cost estimation and DoD program cost experience. One of our goals is to create an online repository of domain reference points that embodies the historical DoD program cost experience.
The repository will include a searchable database of reference points that helps domain experts exercise better judgment during cost estimation. Domain experts will be able to query the reference points using key words based on search technology. Search results will show the key reference points in relation to the domain and technology challenge. The domain expert(s) will then review those reference points before formulating judgement for the current cost estimation exercise. At this point in the project, we are mining reference points from DoD and other open-source data examples.
My colleague, James McCurley, has investigated DoD repositories for raw information that outlined why acquisition programs experience cost and schedule overruns. Our team compiled domain reference points from McCurley's data that identify selected changes associated with cost and schedule overruns. We are categorizing these changes into a set of common change drivers that are rooted in the various sources we have accessed.
One of the first sources we accessed was the U.S. Navy's Probability of Program Success (PoPS) program. The PoPS criteria came from studies of program performance used by the Navy to implement a step-by-step approval process for a program to continue, independent of--but aligned to--the DoD acquisition process. PoPS identified a number of categories of reasons for cost and schedule overruns in government programs.
PoPS was always seen as just one of many sources of programmatic factors that might provide information useful to QUELCE. The PoPS criteria are biased toward programmatic change issues (such as sponsorship, contractor performance, and program office performance) that are of primary concern to DoD sponsors and Program Executive Offices. As expected, when we started this project, we are finding the need to supplement PoPS with more technical change issues, such as those related to system engineering and integration factors.
Many technical change drivers may be seen in the Capability Based Assessment (CBA) activity performed by programs in preparation for the Milestone A decision. The CBA includes the Functional Area Analysis (FAA), Functional Needs Analysis (FNA), and Functional Solution Analysis (FSA). Another early source is the Analysis of Alternatives (AoA). These and other early documents often include information that identifies technical and programmatic uncertainties not captured in the cost estimation process, but which can be incorporated as program change drivers in QUELCE method. Consequently, many technical change drivers are rooted in artifacts that proposed programs must draft prior to Milestone A.
Examples of change drivers we've identified from various sources include:
Applying Our Expert Opinion Approach
In the coming year, our goal is to develop a database with information that supports experts implementing the QUELCE method. We will publish our approach to improving expert judgment and increasing and structuring their involvement in cost estimation using procedures similar to Team Software Process (TSP) scripts. Our goal is to ensure that domain experts have more active involvement with the cost estimation activity. The problem today is that domain expert results are often loosely coupled with the cost estimates. In contrast, QUELCE will facilitate domain experts systematically discussing change drivers and then mapping the change drivers explicitly to the cost driver inputs of traditional cost estimating models and estimating relationships (CERs).
In our approach, the domain expert will be prompted at different points throughout QUELCE to access the reference point database. These activities will consist of just-in-time virtual training for calibration within a given domain. For example, a domain expert may be participating in QUELCE to develop a cost estimate for a new communication system that involves satellite technology. If the domain expert has not recently completed virtual calibration training for that domain, he or she may receive a refresher course consisting of a two- to four-hour online exercise.
Our approach to improving expert opinion will help domain experts during the following three different points in the QUELCE method that depend significantly on expert judgment:
While our work to date has focused on calibrating expert judgment in the DoD cost estimation of program development, our approach could be applied to many situations beyond cost estimation. We envision this approach being used in domains such as portfolio management and strategic planning.
Our research into the QUELCE method for pre-Milestone A cost estimation represents a significant advance by enabling the modeling of uncertain program execution scenarios that are dramatically different from the traditional cost factor inputs of cost estimation models currently employed later in the DoD acquisition lifecycle. By synergizing the latest advancements in proven methods such as scenario planning workshops, cause-effect matrices, BBNs, and Monte Carlo simulation, we have created a novel and practical method for early DoD acquisition lifecycle cost estimation. If you're interested in helping us succeed in these efforts, please let us know by leaving a comment below.
To read the SEI technical report, Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE) please visit www.sei.cmu.edu/library/abstracts/reports/11tr026.cfm.
For more information about Milestone A, please see the Integrated Defense Life Cycle Chart for a picture and references in the "Article Library."
Visit the SEI Digital Library for other publications by Robert