Background: Over the past decade, the U.S. Air Force has asked the SEI's Acquisition Support Program (ASP) to conduct a number of Independent Technical Assessments (ITAs) on acquisition programs related to the development of IT systems, communications, command and control, avionics, and electronic warfare systems. This blog post is the third in a series that enumerates common themes across acquisition programs that we identified as a result of our ITA work. Other themes explored in this series include misaligned incentives, the need to sell the program, and common infrastructure and joint programs. This post explores the third theme in this series, the evolution of "science projects," which describes how prototype projects that unexpectedly grow in size and scope during development often have difficulty transitioning into a formal acquisition program.

The Third Theme: The Evolution of "Science Projects"

A growing theme in acquisition is the increasing prevalence of programs that are often initially called "science projects," and the difficulties associated with evolving such efforts into larger systems and more formal acquisition efforts. The name "science project" refers to a program that starts out small--often as an experimental development or prototype system--so that it can

1. clarify user requirements
2. better understand the problem
3. produce a compelling "proof of concept" to help solicit funding and 'sell' the program

The recurring dynamic of science projects has received little discussion in acquisition literature, even though the pattern is increasingly common. Due in part to urgent demands for new technologies in theaters such as Iraq and Afghanistan, a quarter of the programs that we assessed for the Air Force study mentioned above could be characterized as having begun as science projects.

Many defense programs that started out as science projects ultimately produced important advances in technology that leapfrogged our adversaries and gave American warfighters a critical edge in conflicts around the world. In many cases the ability to quickly develop and deploy a new technology has been key to adapting effectively to rapidly changing conditions and threats. The question is not whether we need science projects--but how we can most effectively--and sustainably--move the technology from a prototype into the hands of the user or warfighter.

Science projects are often initiated--and even frequently managed--by their user community, who are often experts in the relevant domain area but lack expertise in software engineering and management. Some projects may be organized as less formal "in-house" developments, while others are begun using contractors who specialize in building advanced prototype systems. Still others are the product of laboratories or Advanced Concept Technology Demonstration (ACTD) efforts.

In any case, science projects often evolve organically without the structure or investment in design that should be put into building a large and mission-critical software-reliant system. These short-cuts sometimes occur because there isn't yet adequate demand for the capability to justify a formal program. Other times there has been a conscious decision to try to develop it as quickly as possible by "flying under the radar" and avoiding the bureaucratic overhead and cost of the formal acquisition process.

A potential downside with science projects, however, is that these systems can become the victims of their own success. The natural and laudable instinct of commanders to provide valuable new capabilities to their warfighters in the field as quickly as possible can become a threat to the project. The initial prototypes of science projects often grow incrementally without a guiding vision or architecture and are tested in the field, often with very positive initial reactions to the system's new capabilities. Once the value of the system's capability is recognized by warfighters and other users, demand increases quickly, and the warfighters may soon become unwilling to part with these new capabilities. The systems have already incorporated many new features in response to warfighter needs, with the code base becoming increasingly hard to maintain, and bugs being injected as each change is made or new feature is added. Documentation--rarely a strong point of most prototype development efforts--is no longer an option.

In such an environment, science projects are forced to evolve rapidly into full-scale acquisition programs that can reliably deliver a robust production system growth spurt that was neither anticipated nor properly planned for in many cases. It is at this point that many science projects "hit the wall" with a large, mostly undocumented, convoluted, defect-laden, and unmaintainable code base, unable to make progress at their early development rates. Almost any change made to such a system now will have multiple adverse unintended consequences on performance and robustness. The software--like the plumbing in an old house--can only keep working for so long before the entire system must be scrapped and re-implemented.

Sound software engineering practice dictates that when a prototype's objectives have been met, it shouldn't be fielded as an operational system directly. Although many aspects of the prototype can and should be reused, systematic development and quality assurance methods are still needed. This evolution can take many forms, including agile approaches, but all such methods still require software expertise and rigor. Instead, what can happen is that a science project tries to transform its prototype into a full-scale system built on top of an unsound foundation. The system will then often experience problems with robustness, capability, performance, and usability, among other issues.

As major new capabilities must be added to the prototype to satisfy the still-growing user demand, the project's managers see the shortcomings of its evolved design and would like to pause development to re-architect it. Insistent user demand, however, won't allow the project to "go dark" for months--much less years--to do work that will produce little in the way of visible new features for the end users. Even if the government were still inclined to discard the prototype and start new development from scratch, the contractor has an incentive to persuade the government to reuse the prototype software as the platform for the new work, thereby making the contractor more indispensable to the future development effort.

When the technical aspects encounter difficulties, the management aspects do as well. The project infrastructure (the project team size, its processes, the level of software development and management experience, etc.) that may have been adequate for building a prototype can be inadequate for developing a formal production system intended for wide deployment. These mismatches can mean that most aspects of the project are now inappropriate for their new purposes and must either be discarded or substantially changed and expanded.

We're seeing science projects morphing awkwardly and ineffectively into convoluted production systems more frequently due to the desire to deploy new capabilities to the field that demand new technologies at an ever faster pace. Science projects don't have to "hit the wall," as long as the needs to scale up both the system design and project organization are recognized and acted upon. Unfortunately, in an environment of scarce funding and chronic schedule pressure, there is a strong temptation to continue development on the same code foundation, with the original project infrastructure, in the often mistaken hope that it will suffice.

We collectively need to do a better job of working with science projects to help them be more successful in "crossing the chasm" from prototype development efforts to formal acquisition programs. The challenge is to help acquisition staff identify their situation early on, understand the issues involved, and recognize and act upon the need to scale up both the organization and system architecture as needed to be able to complete a successful acquisition effort. In the blog post, Enabling Agility by Strategically Managing Architectural Technical Debt, my colleague, Ipek Okzaya, describes a promising new methodological advance that could help to mitigate some of these problems. Likewise, my colleague Rick Kazman describes strategies for architectural documentation that help make the fundamental design rules of science project software more explicit, thereby helping developers, testers, and maintainers understand the software and work together more effectively. By better understanding how concepts like "technical debt" and "architectural documentation" can help us to more efficiently manage the evolution of software, and apply methods such as refactoring to restructure our code, we are taking important steps toward mastering "science projects."

This is the third post in an ongoing series examining themes in acquisition. The first post explored misaligned incentives. The second post explored the need to sell the program. New installments in the series will be published over the next several months on the SEI blog, where a new post is published every Monday morning.