Posted on by Agile Adoption in Governmentin
More and more, suppliers of software-reliant Department of Defense (DoD) systems are moving away from traditional waterfall development practices in favor of agile methods. As described in previous posts on this blog, agile methods are effective for shortening delivery cycles and managing costs. If the benefits of agile are to be realized effectively for the DoD, however, personnel responsible for overseeing software acquisitions must be fluent in metrics used to monitor these programs. This blog post highlights the results of an effort by researchers at the Carnegie Mellon University Software Engineering Institute to create a reference for personnel who oversee software development acquisition for major systems built by developers applying agile methods. This post also presents seven categories for tracking agile metrics.
An Empirical Approach to Software
Increasingly, the DoD and federal agencies procure software-intensive systems instead of building them with internal resources. However, acquisition programs frequently have difficulty meeting aggressive cost, schedule, and technical objectives. The research reported in this blog is part of our ongoing work to help acquisition professionals in the DoD adopt and use agile software development methods more effectively to overcome these challenges. Our latest research focuses on progress measurement and contractors, which extends our previous research examining selected DoD management and acquisition concerns, including information assurance.
Many program offices in government and military organizations that we support have found that their providers or contractors are adopting agile methods. These program offices have found new ways to deliver software products more rapidly and in smaller increments than is customary in these environments. Program offices struggle with these techniques, however, because they lack experience with the metrics required to gain insight on progress. There is an elaborate infrastructure and a fairly well understood set of definitions for measures traditionally used in that space.
When we examined what is written, trained, and discussed about the term agile, the focus tends to be on a team of seven plus or minus two individuals working together in a self-directed setting. These teams have a keen focus on value to the user and employ an empirical approach, but most of the discussion centers on the small team.
It is important to note that in Scrum, decisions are made based on observation and experimentation rather than on detailed upfront planning. Empirical process control relies on three main ideas: transparency, inspection, and adaptation.
Use of agile methods in the context of major DoD programs is not unprecedented. Until recently, however, publications and training courses have focused too narrowly on the development team. Our findings show that organizations that apply agile methods are doing a good job filling in those abstractions between the small team-focused measurement and what is needed at an enterprise or program level. So, one of the challenges to overcome then, is meeting the needs of large-scale-program-management without violating the environment necessary for a self-directed team to succeed using agile methods.
In many medium-to-large sized organizations, measurement is often conducted at the request of another individual and typically an obligation imposed on one party to benefit another party. Far too often, people asked by team leaders and project managers to provide metrics get defensive. This dynamic may limit the value of agile methods, which are intended to serve as the basis for an empirical approach to software.
This empirical approach involves enacting Edward Deming's plan-do-check-act-cycle, but at a much more immediate and individually focused level. This approach to software involves more frequent conversation among developers. Another hallmark of the approach is that those conversations are very focused on the product itself.
When viewed in the context of the Agile Manifesto and its 12 principles, the best way to demonstrate progress is to demonstrate capability: an actual working product in lieu of an abstraction on paper. Obviously, the product could not have been built without the abstractions, but in the context of agile methods, the focus is on demonstrable results and data collected by the team for its own use.
For agile software development, one of the most important metrics is delivered business value. These progress measures, while observation-based, do not violate the team spirit. Our primary goal with this work was to help program managers measure progress more effectively. At the same time, we want teams to work in their own environment and use metrics specific to the team, while differentiating from metrics that are used at the program level.
The technical report that we published on this topic--Agile Metrics: Progress Monitoring of Agile Contractors, which was co-authored by myself, Suzanne Miller, Mary Ann Lapham, Eileen Wrubel, and Timothy A. Chick--details three key views of agile team metrics that are typical of most implementations of agile methods:
Our research involved interviewing professionals who manage agile contracts who gave us insight from professionals in the field who have successfully worked with agile suppliers in DoD acquisitions.
Based on our interviews with personnel who manage agile contracts, our technical note identified seven successful ways to monitor progress that help programs account for the regulatory requirements that are common in the DoD:
We continue to learn new and inventive ways of demonstrating progress and diagnosing performance from agile implementers. The value of this approach is that it represents a narrative driven by real-world experience.
Through this research, we have also observed that agile teams don't want to wait for data analysis. Future research needs to focus on earlier analysis of data streams. The graphical representations needed to analyze these data streams are not the traditional ones we have seen. The analysis techniques that need to be applied, as well as the available baselines, take on a different form in this context: more near-term, more immediate feedback and the intelligent use of historical baselines.
Acquisition researchers in the SEI's Software Solutions Division have published several technical notes that address different topics of interest to acquisition professionals that are contemplating or are currently using agile methods as part of their acquisition: