Posted on by SPRUCE Seriesin
By Spruce Project
This is the third installment in a series of three blog posts highlighting seven recommended practices for monitoring software-intensive system acquisition (SISA) programs. This content was originally published on the Cyber Security & Information Analysis Center's website online environment known as SPRUCE (Systems and Software Producibility Collaboration Environment). The first two posts in the series explored the challenges to monitoring SISA programs and presented the first five recommended best practices:
This post, which can be read in its entirety on the SPRUCE website, will present the final two recommendations, as well as conditions that will allow organizations to derive the most benefit from these practices.
6. Refresh measures for each new phase/milestone
The program dashboard is a focusing tool to use during program reviews through the life of the program, but the measurements on the dashboard change as the program progresses from phase of work to phase of work. Through the life of the program, the dashboard helps the program manager revisit the same basic questions, but they are not the same specific questions. As mentioned in the first practice, at the beginning of a new development phase, contractors should provide the set of measures they intend to use to report progress. The acquisition team needs to become familiar with these measures and their definitions as described in the third practice.
An example of how measures may change reflects the difference between development activity and operational testing. The measurement of progress changes from deliverables completed to test cases passed. Test cases passed are easier to observe, and so progress is more apparent. Measuring effectiveness of the process is also performed differently. During operational testing, we should observe how long it takes to resolve discovered defects and how much effort is involved in defect tracking. The cycle time for defect management is often directly related to some set of decision processes, and thus its measurement is an indicator of process effectiveness. Such measures of process effectiveness can be supplemented with a count or percentage of "bad test cases." A bad test case is a test that fails whose cause is traceable to an error in the definition of the test--thus representing a process problem rather than a product problem.
7. Set new expectations and negotiate changes to commitments with stakeholders
As a result of the program review or other discussion (fourth and fifth practices), the program manager may conclude that stakeholder expectations need to be reset or commitments need to be changed. The content of a program dashboard, its analysis, and subsequent discussions with the contractor serve as the justification for the reset/change and also serve as the basis for determining what changes to stakeholder commitments are necessary. Giving due diligence to the changes implied by the dashboard helps to sustain stakeholder confidence in the PM's ability to recognize when changes are needed and helps maintain stakeholder interest and support for the program.
Sometimes, the nature of the change is fairly significant. For example: prototypes or simulations are typically performed for any high-risk change to the technology used. The challenge the contractor needs to address is not simply to prove that the technology itself works; rather, the challenge is to prove that the contractor can develop a product that uses the new technology in an integrated product and further that the contractor is prepared to test a product that contains the new technology. It is not unusual to discover that the new technology is much more difficult to apply than expected and that more schedule and perhaps additional resources are required. If the prototype is on the critical path for development and testing, it is virtually certain to delay product delivery. Such significant changes are likely to be reflected in all four quadrants of the dashboard.
Remember also that many of a program's stakeholders have negotiated commitments with their own stakeholders--commitments they are not likely to share with the program manager to retain flexibility in what they can claim they can or can't do when renegotiating commitments. The dashboard evidence the program manager provides about his or her own schedule and resource forecasts, product quality, and the other topics touched on by the four quadrants--but that also are of potentially keen interest to stakeholders--gives the program manager the leverage he or she needs to enter into some potentially complex discussions with stakeholders.
Under what conditions will organizations derive the most benefit from these practices?
When an organization neglects the following factors, the effectiveness of the monitoring practices may be severely limited:
Below is a listing of selected resources where readers can go for more information on the topics covered in this series of posts. We have also added links to various sources to help amplify a point. Please be mindful that such sources may occasionally include material that might differ from some of the recommendations in the article above and the references below.
Richard Crume. "Who Is to Blame When Government Contracts Go Astray?" IACCM, 2008. http://www.iaccm.com/news/contractingexcellence/?storyid=548
Adrian Pitman, Elizabeth K. Clark, Bradford K. Clark, & Angela Tuffley. "An Overview of the Schedule Risk Assessment Methodology (SCRAM)." Journal of Cyber Security & Information Systems 1, 4 (2013). https://www.csiac.org/sites/default/files/journal_files/CSIAC_V1N4_FINAL_2.pdf
Defense Acquisition University. Defense Acquisition Guidebook. DAU, 2013. https://dag.dau.mil/Pages/Default.aspx
Steve McConnell. Software Estimation: Demystifying the Black Art (Best Practices Series). Microsoft Press, 2006.
Software Program Managers' Network. The Program Manager's Guide to Software Acquisition Best Practices. Computers & Concepts Associates, 1998. https://acc.dau.mil/adl/en-US/33409/file/6731/%2316705%20Software%20Best%20Practices%20Initiative.pdf