The extent of software in Department of Defense (DoD) systems has increased by more than an order of magnitude every decade. This is not just because there are more systems with more software; a similar growth pattern has been exhibited within individual, long-lived military systems. In recognition of this growing software role, the Director of Defense Research and Engineering (DDR&E, now ASD(R&E)) requested the National Research Council (NRC) to undertake a study of defense software producibility, with the purpose of identifying the principal challenges and developing recommendations regarding both improvement to practice and priorities for research.

The NRC appointed a committee, which I chaired, that included many individuals well known to the SEI community, including Larry Druffel, Doug Schmidt, Robert Behler, Barry Boehm, and others. After more than three years of effort--which included an intensive review and revision process--we issued our final report, Critical Code: Software Producibility for Defense. In the year and a half since the report was published, I have been asked to brief it extensively to the DoD and the Networking and Information Technology Research and Development (NITRD) communities.

This blog posting, the first in a series, highlights several of the committee's key findings, specifically focusing on three areas of identified improvements to practice--areas where the committee judged that improvements both are feasible and could substantially help the DoD to acquire, sustain, and assure software-reliant systems of all kinds. The "help" is in the form of reduced costs, greater productivity, improved schedules, and lower risks of program failure--and also in enabling the DoD to build systems with much greater levels of capability, flexibility, interlinking, and assurance. The next blog postings will cover some of the lessons learned since the report came out.

Practice Improvement 1: Process and Measurement

Success in developing software-dominated systems requires organizational processes that enable managers and developers to set achievable goals, analyze data, and guide decisions--and to succeed in these processes despite rapid change in operating context and in the technical and infrastructural environment. Advances related to process and measurement help facilitate broader and more effective use of incremental and iterative development methods, which have relatively short process feedback loops. These iterative approaches can better accommodate change and uncertainty. As a consequence, these approaches are commonplace in commercial and enterprise development. But for the DoD, advances in incremental and iterative development methods must account for the typical "arms-length" relationships, common in acquisition programs, that exist between contractor development teams and government stakeholders.

Incremental development practices enable continuous identification and mitigation of engineering risks during a system's development process. Engineering risks pertain to the consequences of particular choices made within an engineering process--the risks are high when the outcomes of immediate project commitments are consequential, hard to predict, and apparent only well after the commitments are made. Engineering risks may relate to many different kinds of engineering decisions--most notably architecture, quality attributes, functional characteristics, and infrastructure choices.

When managed properly, incremental practices can enable successful innovative engineering without increasing the overall programmatic risk related to completing engineering projects, such as managing stakeholder expectations and triaging priorities for cost, schedule, capability, quality, and other attributes. Incremental practices help identify and mitigate engineering risks earlier in system lifecycles than traditional waterfall approaches--the feedback is sooner, and so the costs and consequences are lower. These practices are enabled through the use of diverse techniques, such as modeling, simulation, prototyping, and other means for early validation--coupled with extensions to earned-value models that measure and acknowledge the accumulating body of evidence in support of program feasibility. Incremental methods include iterative approaches (such as Agile), staged acquisition, evidence-based systems engineering, and other methods that explicitly acknowledge engineering risks and their mitigation.

The committee found that incremental and iterative methods are of fundamental significance to innovative, software-reliant engineering in the DoD, and they can be managed more effectively through improvements in practices and supporting tools. The committee recommended a diverse set of improvements related to advanced incremental development practice, supporting tools, and earned-value models.

Practice Improvement 2: Architecture

In software-reliant DoD systems, architecture represents the earliest and often most important design decisions--those that are the hardest to change and the most critical to get right. Architecture is the principal way we address requirements related to quality attributes such as performance, security, adaptability, and the like. Architectural design also embodies expectations regarding the various dimensions of variability and change for a system. When architecture design is successful, system quality is more predictable, and change is more likely to be accommodated through smaller increments of effort rather than through wholesale restructuring of systems.

Advances related to architecture practice thus contribute to our ability to build systems with demanding requirements related to quality attributes, interlinking, and planned-for flexibility.

Software architecture techniques and tools model the structures of a system that comprises software components, externally visible properties of those components, and relationships among the components. Architecture thus has both structural and semantic aspects--it is not just about how components interconnect. Good architecture entails a minimum of engineering commitment that yields a maximum of business value. Architecture design is thus an engineering activity that is separate, for example, from standards-related policy setting and the certification of commercial ecosystems and components.

For complex innovative DoD systems, architecture definition embodies planning for flexibility--defining and encapsulating areas (such as common operating platform environments and cyber-physical systems) where innovation, change, and competition are anticipated. Architecture definition strongly influences diverse quality attributes, ranging from availability and performance to security and isolation. It also embodies planning for the interlinking of systems to form systems-of-systems and ultra-large-scale systems and for product line development enabling encapsulation of individual innovative elements of a system.

For many innovative DoD systems it is essential to consider architecture and quality attributes before making too many specific commitments to functionality. This may seem backwards from the usual model, of putting functional requirements first. But the engineering reality is that architecture includes the earliest and typically the most important design decisions: those engineering costs that are the hardest to change later. Early architectural commitment (and validation) can therefore often yield better project outcomes with less programmatic risk.

The committee found that in highly complex DoD systems with emphasis on quality attributes, architecture decisions may dominate functional capability choices in overall significance. The committee also noted that architecture practice in many areas of industry is sufficiently mature for the DoD to adopt. The committee recommended that the DoD more aggressively assert architectural leadership, with an early focus on architecture being essential for systems with innovative functional or demanding quality requirements.

Practice Improvement 3: Assurance and Security

A significant--and growing--challenge for DoD systems is software assurance, which encompasses diverse reliability, security, robustness, safety, and other quality-related and functional attributes. The weights given these various attributes are often determined by modeling hazards associated with operational context, including potential threats and the penalties of system failure. Software assurance is very expensive--the process of achieving assurance judgments, regardless of sector, is generally recognized to account for approximately half the total development cost for major projects. Advances related to assurance and security would therefore facilitate greater mission assurance for systems at greater degrees of scale and complexity.

Advances in assurance and security are particularly important to the rich supply chains and architectural ecosystems that are increasingly commonplace in modern software engineering. The growing reliance on software by the DoD has increased the functional capability of all kinds of systems, as well as a growth in the interconnectedness of systems and the extent of potential for rapid adaptation of systems. With this growth has come a dependence of DoD software-reliant systems on increasingly complex, diverse, and geographically distributed supply chains. These supply chains include not only custom components developed for specific mission purposes, but also commercial and open-source ecosystems and components, such as the widely used infrastructures for web services, cloud computing environments, mobile devices, and graphical user interaction. This places emphasis on composition and on localizing points of trust within a system.

In addition to managing overall costs, the DoD faces many challenges for assurance relating to technology, practices, and incentives, including:

• The arms-length relationship between contractor development teams and government stakeholders also complicate the creation and sharing of information necessary to make assurance judgments. This type of relationship can lead to approaches that focus excessively on post hoc acceptance evaluation, rather than on the emerging practice of building evidence in support of an overall assurance case.
• Modern systems draw on components from diverse sources, implying that supply-chain and configuration-related attacks must be contemplated, with "attack surfaces" existing within an overall application, and not just at its perimeter. The consequence of this trend is that evaluative and preventive approaches should ideally be integrated throughout a complex supply chain. A particular challenge is managing black box components in a system (this issue is addressed in the full report).
• The growing role of DoD software in warfighting, protection of national assets, and the safe guarding of human lives creates a diminishing tolerance for faulty assurance judgments. The Defense Science Board notes that there are profound risks associated with the increasing reliance on modern software-reliant systems: "this growing dependency is a source of weakness exacerbated by the mounting size, complexity, and interconnectedness of its software programs."
• Losing the lead in the ability to evaluate software and prevent attacks can confer advantage to adversaries with respect to both offense and defense. It can also force the DoD to restrict functionality or performance to a level such that assurance judgments can be achieved more readily.

The Defense Science Board also found "it is an essential requirement that the United States maintain advanced capability for 'test and evaluation' of IT products. Reputation-based or trust-based credentialing of software ('provenance') needs to be augmented by direct, artifact-focused means to support acceptance evaluation." Achieving this capability is a significant challenge due to the rapid advance of software technology generally, as well as the increasing pace by which potential adversaries are advancing their capabilities. This challenge--coupled with the observations above regarding software innovation--provides an important part of the rationale for the committee's recommendation that the DoD actively and directly address its software producibility needs.

The committee found that assurance is facilitated by advances in diverse aspects of software engineering practice and technology, including modeling, analysis, tools and environments, traceability and configuration management, programming languages, and process support. The committee also found that simultaneous creation of assurance-related evidence with ongoing development has high potential to improve the overall assurance of systems. The committee recommended enhancing incentives for software assurance practices and production of assurance-related evidence throughout the software lifecycle and through the software supply chain for both contractor and in-house developments.

Looking Ahead

The next blog posting in this series will focus on lessons learned in the many interactions subsequent to the publication of the NRC Critical Code report. I will also discuss what these lessons signify for developing software strategy for the DoD, in general, and the SEI, in particular.

Additional Resources

This posting is an excerpted, edited copy of an article that Bill Scherlis wrote for The Next Wave, "Critical Code: Software Producibility for Defense," which was published in Volume 19, No. 1 (2011). To request copies of the journal, please send an email to tnw@tycho.ncsc.mil.

To download a PDF of the report Critical Code: Software Producibility for Defense, go to
www.nap.edu/catalog.php?record_id=12979.

To download a PDF of the Report of the Defense Science Board Task Force on Defense Software (2000), go to http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA385923