Department of Defense (DoD) programs have traditionally focused on the software acquisition phase (initial procurement, development, production, and deployment) and largely discounted the software sustainment phase (operations and support) until late in the lifecycle. The costs of software sustainment are becoming too high to discount since they account for 60 to 90 percent of the total software lifecycle effort.

Moreover, in an era where DoD new-start programs are being reduced in favor of prolonging legacy systems, significant software sustainment cost increases are themselves unsustainable. The growing expense and prolonging of legacy systems motivates the need for greater discipline and attention on defining and applying appropriate methods and technologies to improve sustainment capabilities and efficiencies. This SEI blog posting--the first in a two part series--summarizes key software sustainment challenges faced by DoD; the subsequent post describes R&D activities conducted by the SEI to address some of these challenges.

Software sustainment involves coordinating the processes, procedures, people, information, and databases required to support, maintain, and operate software-reliant aspects of DoD systems. DoD software sustainment needs and practices are shaped by various trends, including

• rapid performance advances driven by Moore's Law and associated hardware innovations, which accelerate technology refresh cycles,
• the prevalence of commercial-off-the-shelf software technologies and practices, which commoditizes the market for software engineers with modern skills, but creates gaps for projects that need staff with expertise in older technologies,
• the need to adapt software to address diminishing manufacturing sources stemming from the loss of producers or suppliers of hardware used in DoD systems,
• the challenges of modernizing and recapitalizing legacy DoD systems in a constrained budget environment that emphasizes greater efficiency and productivity in defense spending, and
• the repurposing of systems to meet new threats and increasing requirements for interoperability.

The confluence of these trends impacts the workload, risk, and cost of acquisition program sustainment processes and professionals. These trends also contribute to the growth in total ownership costs across program lifecycles.

Newer DoD systems rely more on software than older systems did, so the demands on software sustainment organizations are increasing as current generations of DoD systems transition from production to sustainment. For example, the percentage of avionics specification requirements involving software control has risen from approximately 8 percent of the F-4 in 1960 to 45 percent of the F-16 in 1982, 80 percent of the F-22 in 2000, and 90 percent of the F-35 in 2006. This growing reliance on software now affects most aspects of DoD systems, including mission data systems, radars/sensors, flight/engine controls, communications, mission planning/execution, weapons deployment, test infrastructure, program lifecycle management systems, and software integration labs.

Not only are we dealing with a growing software base, but also the constantly-evolving environment in which software runs. For example, although software does not wear out, firmware and hardware become obsolete, thereby requiring software modifications. Likewise, upgraded capability must be integrated into existing systems and software defects and performance bottlenecks must continually be identified, fixed, and optimized to provide full functionality. It should therefore not be surprising that the DoD expends an increasing amount of time and effort sustaining software, often much more than was originally anticipated due to uncertainties during initial program cost estimation. While programs can take funds from later phases to cover development overruns, the sustainment phase has nothing after it to prey upon but itself!

High software sustainment costs occur for various reasons. For example, functionality originally provided by hardware may be replaced by software (e.g., fly-by-wire), thereby increasing software sustainment costs. Periodic software upgrades and enhancements throughout the lifecycle of DoD systems may also result in unanticipated increases in sustainment costs. Moreover, costly and time-consuming effort is required by software maintainers to understand the original design and carefully make changes to avoid degrading the integrity of the design or negatively impacting key quality attributes. In addition, software scale and complexity are growing significantly to meet the expanded threat spectrum, which drives sustainment costs up.

As sustainment costs have increased, the DoD is struggling to support all its legacy systems. Economic strategies for understanding and addressing these rising costs are affected by a key difference between DoD weapon system platforms and the software running in those platforms (example of weapons systems platforms include the physical airframes, hulls, chassis, and their associated parts like engines, weapons, sensors, and computing/communication units). The DoD has historically viewed sustainment from a weapon-system-platform perspective--where physical manufacturing and handling wear-and-tear represent a significant expense. From this perspective, sustainment costs are primarily a function of the number of weapon system platforms and parts. The DoD has traditionally handled these burgeoning costs by shrinking its inventory (for example, by retiring and/or reducing the numbers of aging aircraft, ship, and vehicle platforms).

This traditional approach worked when sustainment costs were largely a function of weapon system platform and part manufacturing costs. In contrast, software sustainment has essentially no manufacturing or wear and tear expenses. As a result, software sustainment costs are primarily a function of the number of software variants. For example, a particular weapon system platform family (such as a class of ships, planes, or vehicles) may have scores of software variants reflecting different sensor, processing hardware, operating system, and network/bus configurations; different algorithms; and different security profiles allowed for use by customers from different countries. Sustaining all these variants impacts the time and effort required to assure, optimize, and manage system deployments and configurations throughout the lifecycle.

As software variability grows--as it inevitably does in legacy systems unless a concerted effort is made to rein it in--it becomes increasingly hard to avoid adding unnecessary variability, reimplementing variation mechanisms more than once, selecting incompatible or awkward variation mechanisms, and missing required variations. The DoD is therefore facing "sticker shock" since software sustainment costs are unlikely to decrease by shrinking its inventory alone since roughly the same level of software sustainment is still needed, regardless of whether there are 100 or 10,000 hardware platforms. What the DoD needs are different strategies for understanding and alleviating rising software sustainment costs--such as sustainment strategies based on managing software commonality and variability via software product lines.

In addition to the challenges above, the DoD also faces challenges with recruiting, developing, and retaining its software sustainment workforce. For example, although the DoD needs efficient and productive software sustainers, this specialty is often not viewed as exciting or innovative as green-field developments, so key research and development challenges remain unresolved. Effective sustainment also requires engineers who have expertise in older languages, operating platforms, and tools combined with deep domain and software architect knowledge. This combination tends to reside in more experienced members of the DoD workforce, so retaining and replenishing this cadre of software engineers is important. Fortunately, some software engineers dedicate their careers to sustaining software, though their efforts are often not as well recognized (or funded) by the DoD relative to their importance as the inventory of DoD systems continues to age.

This blog posting just scratched the surface of the technical and non-technical challenges associated with sustaining software-reliant DoD systems. Another vexing, non-technical challenge is that DoD contracts often fail to procure source code, necessary licenses and technical data rights, and technical data on design artifacts, testing facilities, and procedures during the acquisition process. The failure to procure this material complicates software sustainment activities and increases total ownership costs over program lifecycles. The second part of this series describes R&D activities the SEI is conducting to address some of the technical challenges presented above.

Additional Resources:

More information about sustaining software-reliant DoD systems is available below.

To read about the National Research Council's Aging of U.S. Air Force Aircraft report, please visit www.nap.edu/catalog.php?record_id=5917.

To read about the Air Force Science Advisory Board's Sustaining Aging Aircraft report, please visit www.airforce-magazine.com/SiteCollectionDocuments/Reports/2010/December%202010/Day29/Aging_TOR_2011.pdf.

To read about the National Research Council's Critical Code: Software Producibility for Defense report please visit www.nap.edu/openbook.php?record_id=12979&page=R1.

To read the SEI technical report, Sustaining Software-Intensive Systems, please visit
www.sei.cmu.edu/library/abstracts/reports/06tn007.cfm or to see a presentation on Sustaining Software Intensive Systems - A Conundrum please visit www.dtic.mil/ndia/2005systems/wednesday/lapham.pdf.