While agile methods have become popular in commercial software development organizations, the engineering disciplines needed to apply agility to mission-critical, software-reliant systems are not as well defined or practiced. To help bridge this gap, the SEI recently hosted the Agile Research Forum. The event brought together researchers and practitioners from around the world to discuss when and how to best apply agile methods in mission-critical environments found in government and many industries. This blog posting, the fifth and final installment in a multi-part series highlighting research presented during the forum, summarizes a presentation I gave on the importance of applying agile methods to common operating platform environments (COPEs)that have become increasingly important for the Department of Defense (DoD).

The first half of my presentation motivated the need for COPEs that help collapse today's stove-piped, software-reliant DoD system solutions to decrease costs, spur innovation, and increase acquisition and operational performance. Since this material has appeared in my first SEI blog posting on COPEs, I'll skip it in this posting and focus on the second half of my presentation, which discussed how applying agile methods can encourage more effective collaboration between users, developers, testers, and certifiers of COPEs to help the DoD build integrated, interoperable, and affordable software-reliant systems more successfully.

What's Taking So Long to Achieve the Promise of COPEs?

Decades of public and private research and development investments--coupled with globalization and ubiquitous connectivity--has enabled information technology to become a commodity, where common off-the-shelf (COTS) hardware and software artifacts get faster and cheaper at a remarkably predictable pace. During the past two decades, we've benefitted from the commoditization of hardware and networking elements. More recently, the maturation and widespread adoption of object-oriented programming languages, operating environments, and middleware is helping to commoditize many software components and architectural layers. Despite these technology advances, however, developing affordable and dependable COPE-based solutions remains elusive for the DoD. There are a number of reasons for this, including

• When DoD acquisition programs have tried to apply COPEs, they've tended to spend a lot of time building common software infrastructure, which consists largely of various layers of middleware. This process leads to something called the serialized phasing problem, where developers spend so much time building the infrastructure, that it may take years (often hundreds or thousands of person years) to develop this infrastructure. Meanwhile, application developers sit idle, not knowing precisely what the characteristics of the infrastructure will be. As a result, by the time the infrastructure has matured enough to support application development, the teams often discover the infrastructure provides inappropriate quality-of-service (QoS) or functionality. If this discovery occurs late in the lifecycle (e.g., during system integration, which is all too common in large DoD acquisition programs) it's extremely costly to remedy.
• A related problem faced by acquisition programs is the length of time it takes to get projects under contract via traditional contracting models. While this glacial contracting pace is not unique to COPEs, it greatly exacerbates the serialized phasing problem outlined above. In particular, if contract delays become excessive, infrastructure developers won't have sufficient time to build and test the common software infrastructure thoroughly. If the common software infrastructure is not built and tested properly, application developers will end up wrestling with many defects and bottlenecks they are ill-equipped to handle. Problems stemming from serialized phasing that are caused by contract delays will result in even further delays in application delivery.
• Classic DoD waterfall models assume that requirements can be largely defined early in the lifecycle. When some of these requirements change--as they certainly will as COPEs are reapplied in different contexts--it becomes quite expensive for the government to request the change orders needed to make the modifications. Without more streamlined and flexible lifecycle and contracting models, the inevitable changes to COPEs will be prohibitively expensive, thereby obviating the goal of cost savings achieved by sharing common software.
• There's a long-term trend in the DoD towards adopting COTS technologies for hardware and software. Although many COTS products are well-suited for mainstream commercial applications, they're often not as well-suited for certain types of DoD systems, especially mission-critical weapons systems. The challenge for COPE developers is to ensure that they don't base their common software infrastructure on COTS products that work well when applied in contexts where an 85 percent solution is sufficient (where dropping an occasional call or manually refreshing a hanging web page is acceptable) in a DoD weapons system with more stringent QoS requirements. In these mission-critical environments--where the right answer delivered too late becomes the wrong answer--many COTS products are too big, too slow, or too unreliable to serve as the basis of mission-critical COPEs. If DoD common software infrastructure is built naively atop a house of sand, applications will be hard-pressed to provide the end-to-end QoS that's expected of them in hostile settings.
• Another challenge facing developers of COPEs is overly-adhering to ossified standards and reference architectures that made sense a decade or so ago, but which failed to keep pace with technology advances. Unlike mainstream commercial systems--where technologies are often refreshed every couple of years--a longer-term perspective is needed to develop COPEs for DoD weapons systems. Likewise, it's essential to integrate new technologies into COPEs to keep pace with advances in hardware, software, platforms, and requirements. We also need software architectures that are more flexible and resilient than simply adhering to standards that become obsolete. The crucial issue here is not new standards, but the capability of coming up with new architectures and new standards that themselves can be refreshed and reapplied with minimal impact over long periods of time.

How Agility Helps Achieve the Promise of COPEs

At the heart of the problems described above is a lack of a holistic approach that balances key business, managerial and technical drivers at scale. The last part of my presentation discussed key success drivers for COPE initiatives that proactively and intentionally exploit commonality across multiple DoD acquisition programs and outlined ways in which agility helps to make those drivers more effective. In my experience working at the SEI, DARPA, and Vanderbilt University's Institute for Software-Integrated Systems for the past several decades, successful COPE efforts require a balance of the following drivers:

• Business drivers, which focus on achieving effective governance and broad acceptance of the economic aspects of COPEs. Examples include managed industry/government consortia, agile contracting models, and effective data rights and licensing models.
• Management drivers, which focus on ensuring effective leadership and guidance of COPE initiatives. Examples include mastery of agile lifecycle methods, strong science and technology connections to reduce technical risk, and of course a solid understanding of the critical role of software for the DoD.
• Technical drivers, which focus on the foundations of COPE development. Examples include systematic reuse expertise, agile architecture expertise, and automated conformance and regression test suites.

As outlined above, agility can be applied to help each of these drivers. For example, agility can be applied to expedite contracting, which is an important business driver. Getting task, delivery, and change orders in place quickly via agile contracting models (such as Indefinite-Delivery, Indefinite-Quantity (IDIQ) contract vehicles) helps to mitigate serialized phasing problems. Likewise, agile contracting methods can also help ensure that the people who are building the systems and the people who are doing the acquisition triage and align their priorities so that the contract supports the needs and considerations of key stakeholders.

Agility can also be applied to manage COPE lifecycles more effectively, which is a key management driver. Common software infrastructure development efforts work best when there are close, interactive feedback loops between people building the applications and the systems engineers and the software engineers and developers building the infrastructure. Without these feedback loops, it's easy to develop many reusable artifacts that aren't useful and won't be applied systematically. An agile approach is thus essential to

• enable close cooperation between users, developers, testers, and certifiers throughout the lifecycle to ensure the necessary COPE capabilities are delivered rapidly and robustly
• avoid integration "surprises" where things tend to break or underperform at unexpected times late in the lifecycle when its more expensive to fix problems

We've observed in recent years that rolling out incremental deliveries of a COPE capability every 4 to 8 months helps application developers establish a battle rhythm of knowing when to upgrade and when to leverage what's in the common software infrastructure. This tight spiral avoids long and fragile serialized phasing lifecycles, which is something that agile methods do a good job of from a management perspective.

Finally, agility can be applied to ensure architectural flexibility, which is a crucial technical driver. Long-lived, software-reliant DoD systems need software architectures that are change tolerant. Inevitably standards will evolve; hardware will get better, faster, and cheaper; and software programming languages, operating systems, and middleware will all evolve over time. It is therefore essential to devise ways of "future proofing" COPE architectures and using technologies, techniques, and methods (such as the technical debt work that Ipek Ozkaya discussed at the Agile Research Forum) when making decisions about when to reengineer and when to refactor. These decisions should be based on empirical data and evidence, rather than relying on forecasts or legacy commitments that become obsolete and ossified over time.

Equally important is the ability to select COTS technologies for use in COPEs that have appropriate QoS capabilities for the ways in which they are applied to particular missions. Some COTS- and standards-based products work well in mission-critical contexts, whereas others don't. The choice of which ones to use should be driven as much as possible through trade studies, empirical analysis, and various types quantitative analysis, as opposed to the latest techno-fad.

The following table summarizes how agility helps resolve the COPE challenges discussed above:

The Road Ahead for COPE Agility

One reason for the spotty track record of success with COPE-based systems is that the DoD hasn't taken a holistic view of the way these types of systems are built. Existing brittle and proprietary stovepiped approaches to acquisition systems do not address the cost efficiency and cyber security challenges the DoD are wrestling with, nor do they help to deploy software and new technologies to the field rapidly. Moreover, managing the production of these systems via waterfall processes is simply not an effective way forward, especially in the shadow of sequestration.

Developing COPEs is achievable and valuable, but it's not easy. Agility in business, management, and technical dimensions is essential, but it's also no panacea. Additional research and engineering investment is needed to devise the appropriate methods, tools, and techniques that will enable agility at-scale, which is a key theme that we've emphasized throughout the SEI Agile Research Forum.

Finally, we need your help. Achieving success with COPEs for the DoD isn't something that can be done by any one group, institute, or government program. The SEI needs to help bring together researchers, developers, and managers from academia, industry, and government to conduct the appropriate work to help reduce risk and ensure the success of current and planned COPE initiatives. We also need to work closely with academia and industry to train the workforce and identify key requirements. Likewise, we need to work with government to ensure effective oversight and acquisition dynamics to reduce the cycle time needed to acquire new systems, insert new technology into legacy systems, and sustain software-reliant systems at a lower cost over their lifecycles and across the DoD enterprise. The stakes are high, and now is the time to make a difference!

Additional Resources

To learn more about the SEI's work on common operating platform environments, please visit
http://insights.sei.cmu.edu/archives.cfm/category/common-operating-platform-environments-copes

To learn more about the SEI's work on agile methods at-scale, please visit the SEI Agile Research Forum webinar site at http://www.sei.cmu.edu/go/agile-research-forum/.