The Government Accountability Office (GAO) has frequently citedpoor cost estimation as one of the reasons for cost overrun problems in acquisition programs. Software is often a major culprit. One study on cost estimation by the Naval Postgraduate School found a 34 percent median value increase of software size over the estimate. Cost overruns lead to painful Congressional scrutiny, and an overrun in one program often leads to the depletion of funds from another. This post, the first in a series on improving the accuracy of early cost estimates, describes challenges we have observed trying to accurately estimate software effort and cost in Department of Defense (DOD) acquisition programs, as well as other product development organizations.

Periodically, the SEI is called to review a program's software estimate usually because two independently generated estimates are far apart - sometimes by a factor of 10 or more. Such disparate results will not pass any of the official milestone reviews and can delay program startup by several months. The frequency of this problem increased with 2008 changes in acquisition regulations by the DOD that require a full life-cycle cost estimate for review at Milestone A, which occurs at the end of the Material Solution Analysis Phase (For more information about Milestone A, see the Integrated Defense Life Cycle Chart for a picture and references in the "Article Library"). Formal acceptance of the Milestone A review is signified by the Acquisition Decision Memorandum (ADM). The ADM is required by law in order to issue a Request for Proposal (RFP) to contract for the Technology Development Phase(TDP).

Before describing our approach, which we will do in the second post in this series, it's important to understand and evaluate the traditional methods of preparing estimates for acquisition programs. Typically, estimators review the available program information, which includes the following documents:

• An Analysis of Alternatives (AOA) describing the proposed solution.
• An Initial Capabilities Document (ICD) and various strategy documents supporting a RFP.
• Preliminary plans for systems engineering, test, and evaluation and similar early planning documents.

Estimators then seek out available cost estimation relationships (CERs) and data from past programs. The estimators must determine which analogies make the most sense. They then apply their expert judgment to prepare a single "most likely" value for size, cost and schedule for each major subsystem, which is often called a "point estimate."

After computing a point estimate, the estimators then add a range, say +/- 25 percent, to the point estimate to account for future changes. These estimates and the background information are delivered to the service (e.g., Army, Navy, Air Force) cost estimation center and the Cost Assessment & Performance Evaluation (CAPE) office for independent review. The various estimates and plans become the content for review by the Milestone Decision Authority (MDA), which determines readiness for the TDP and issues the ADM.

Reviewers of the estimate will make a careful examination of the assumptions made by the estimators. Consequently, the program estimate must reflect possibilities for future program change and must provide ranges for possible costs and schedule duration. Potential changes in technology, program structure, mission, and contract must be considered simultaneously. If the estimates are not reasonably close, the MDA is not likely to approve.

In our experience, estimators and reviewers of estimates who use traditional methods of cost estimation are presented with the following challenges because the nature of the information available prior to Milestone A does not correspond well to the input required for these methods:

• The program will not have a detailed requirements document, making it hard to estimate scale or size.
• Staff may not know the specific technologies that will be needed, meaning they may not know the tasks required and the number of trade studies needed.
• The program manager does not know who will perform the development work, so productivity cannot be estimated accurately.
• Estimators may not know what skills will be required, which makes it hard to determine staff training requirements.

Our goal is to develop a method that overcomes the challenges with traditional estimation methods by identifying and expressing the potential changes and ranges of estimating inputs in terms of a probability model. Automation is a key factor in the model so that re-estimating can be quickly performed as the program discovers which changes become realities and which can be ignored. My next posting will describe research that the SEI's Software Engineering Measurement & Analysis initiative is conducting to improve the accuracy of early estimates (whether it's a DOD acquisition program or commercial product development work) and ease the burden of additional re-estimations during the program lifecycle.

Additional Resources:
For more information about the Software Engineering Measurement & Analysis Initiative, please visit
www.sei.cmu.edu/measurement/index.cfm