Software sustainment involves coordinating the processes, procedures, people, information, and databases required to support, maintain, and operate software-reliant aspects of DoD systems. The 2011 book Examination of the U.S. Air Force's Aircraft Sustainment Needs in the Future and its Strategy to Meet Those Needs states

The Air Force is concerned that the resources needed to sustain its legacy aircraft may increase to the point where they could consume the resources needed to modernize the Air Force.

With millions of lines of code riding on aircraft and automobiles, the cost of software sustainment is increasing rapidly. Several studies show that the cost of sustainment is already as much as 70 percent of the total cost for the life of the software. All the armed services face similar challenges, including deciding how to improve the efficiency and productivity of sustainment organizations and how much should be invested in these improvements. This blog post describes an SEI research initiative aimed at developing an economic model to help anticipate costs and postpone the potential tipping point when sustaining current products is less attractive than replacing legacy systems.

Balancing Stakeholders and Resources

The software sustainment problem is particularly complex for the Department of Defense (DoD) because funding decisions involve an understanding of tensions between three different perspectives:

• operational need (warfighter view)
• management of the portfolio (materiel view)
• capability and capacity of the sustaining organization (process, skills, tools and people)

Our research is motivated by the need to help the DoD make decisions about allocating resources between sustainment work (supporting the warfighter) and improving the performance of sustainment organizations in ways that optimize long-term value to the armed services. Performance improvements are needed in many situations, including

• new test kits to support deployment of new radar or electronic warfare capabilities
• software analysis tools to analyze existing code to help developers learn a legacy system and accelerate their understanding of supported products
• tools supporting automated software testing to accelerate the testing process and assure test coverage of supported products
• training for engineers when upgrades to an existing system employ new processors, operating systems, or programming languages

This SEI research initiative is developing an economic model to support decisions about allocating investments in various performance improvement alternatives. The model will analyze various factors, such as demand for sustainment, capacity of an organic workforce to perform sustainment activities, as well as timing of funding in terms of its impact on long-term costs and readiness of aircraft fleets.

I, along with fellow researchers Sarah Sheard, Andrew Moore, William Nichols, and Mike Phillips, have spent the last year exploring several issues related to software sustainment. Part of our work included developing a systems dynamics model to study how funding decisions and the timing of implementation of changes in sustainment organizations affect the performance of both sustainers and warfighters. We theorized that small differences in funds and timing can have large impacts on performance. This type of system dynamics model uses stocks and flows to represent sustainment performance over time.

Foundations of Our Approach

Jay Forrester, a professor at Massachusetts Institute of Technology, founded system dynamics in the 1950s. Systems dynamics modeling studies the changes in many interrelated variables. Since its inception, system dynamics has been used as a modeling approach in the study of economics and organizations.

With many factors changing simultaneously, simpler economic models (such as return-on-investment and net present value) are insufficient. The interaction of the many inputs to sustainment work can cause emergent effects, such as a sudden and dramatic change in ability to meet demand. Through modeling and analysis research, we are looking for the minimum amount of data that can be used to forecast a sudden and dramatic change (which is also known as a "tipping point"). Forecasting the tipping point gives decision makers time to take action before a problem becomes intractable.

We seek to answer the following questions through our research:

1. Does the model actually show the expected tipping point behavior either in the performance of the sustainment organization or the demand for system improvement?
2. What data provides an indication of growth in demand for sustainment on a particular acquisition program? If we can identify the needed data, will it be possible to collect it and do the necessary analysis?
3. Do the models provide actionable information to decision makers in a timely manner? For example, does reallocating some funding from the sustainment work to the development of the workforce (test kits, process changes, technology training) help reduce the cost of sustainment?
4. What measure of warfighter readiness correlates to the predictive factors in the model?

Through this approach, we aim to help DoD acquisition programs better plan their financial investments to ensure long-term software sustainment and deliver the best value for taxpayer dollars.

While the construction of a model that exhibits the expected changes in the performance of both mission and sustainment is a necessary step, much of our research will focus on sources of data. Real data collected from real programs will be needed to calibrate the model for making real decisions, including

• potential data collection points within the sustainment processes,
• such as demand for sustainment work, rate of hiring and attrition, rate of delivery of, sustainment work, and availability of skilled staff
• potential opportunities to measure warfighter readiness or system use
• such as fleet availability, successful mission performance, and error rate in operation
• standards for applying data collection across different kinds of products such as airplanes, satellites, and communications systems

The Systems Dynamics Model

The basic goal of a simulation model is first to represent the normal behavior of a system and then stimulate it with a new input to see how the responses change.

The model that we have developed represents the behavior of the different players in the sustainment process, including the warfighter, the technical capability of the sustainment organization, and the capacity of the sustainment organization to deliver the work. We are testing the system response to various scenarios, such as

• Threat Change. An external change (such as a new threat to the warfighter) results in a request to update the system capability. This request means the sustaining organization will have to perform both product and process changes; the development process and testing may need to change as well. The changes often require some funding to re-equip the facility and re-train the workforce. Our systems dynamics model helps decision makers analyze the effect if funding for this improvement is delayed.
• Support Technology Change. The sustainment organization decides to improve its own throughput and adopts new processes to "do more with less." Typically the change is also in response to new quality goals. In this case our model helps codify the effect on sustainment capability and capacity and therefore on operational performance.
• Workforce Changes. Sequestration effectively decreases the staff available to sustainment organizations by 10 percent to 20 percent. How does this decrease affect a sustaining organization's ability to meet its sustainment demand? Does it affect aspects of the warfighter mission as well?

Our current model of sustainment consists of five basic process loops:

1. Mission Demand. This loop represents system deployment and use in theater. With successful application, demand for additional missions and increased capability causes an increase in sustainment support required. When missions are less successful or systems cannot be deployed in time, demand for the system decreases and funding support wanes.
2. Sustainment Work. The process of sustainment typically takes a system off-line as it is enhanced, repaired, and redeployed. Some of the requests for sustainment include requests for additional capability.
3. Limits to Growth. This loop shows how the capacity and capability of a sustainment organization limit the rate of completion of sustainment work. As these limits begin to extend the time required to redeploy, the long-term effect may be a reduction in demand or a switch to using an alternate platform.
4. Work Bigger. A sustainment organization may attempt to meet sustainment demand by employing overtime or extra contract employees. Either of these approaches may work for a short time or a small additional cost, but they stress the organization and quickly reach limits of effectiveness. The organization can hire staff, but it must also allow time for training and acculturation of new hires to meet performance objectives.
5. Work Smarter. The sustainment organization invests in new capabilities (skills, tools, and processes) and possibly additional resources (people and facilities) to improve capacity for sustaining work.

Each scenario outlined above entails several decisions and stimulates response curves from the model. The response curves help decision makers forecast how deferring decisions or reallocating resources affect both warfighters and sustainment organizations. We will consider our systems dynamic model "good" if decision makers believe they are able to make faster decisions and if the data from the model makes it easier to get sponsor support for the decisions.

Initial Findings and Future Work

A recent study by the U.S. Army provided us with information confirming the importance of understanding the cycles of demand for sustainment work. From that study, we identified three patterns of release in the software sustainment lifecycle:

• The basic fix pattern mainly focuses on low-level defects and addresses software breaks, which occur frequently (at least once a year).
• The block release pattern mainly focuses on enhancements to address users who have discovered new opportunities to use the system but need some additional functionality to employ it efficiently. The resulting block releases can include additional features for which there are plans and budgets. These sustainment efforts are not patches or quick fixes; they typically occur every year to two years.
• The modernization pattern addresses a significant change in technology or in the pattern of use, such as adding electronic warfare capabilities to an aircraft or accessing an additional external system that hasn't been accessed previously. These sustainment efforts occur about every four to seven years. This third type of release is also often three to four times more expensive than the block release.

The modernization release is often a response to some technology change. The timing (at least four years) has an interesting correlation. Moore's Law suggests the cost-performance of processors doubles every 18 months. A modernization release every four years skips about two processor generations. A modernization release every seven years skips about six generations.

Modernization releases require the sustaining organization to develop new practices and tools and to retrain some existing personnel. Funding for these internal changes is often hard to justify and may be delayed. These delays, in turn, can cause the sustaining organization to fall so far behind the technology curve that a new acquisition contract is often required. If a contractor does win the development contract, the organic sustainment organizational capability may decline while the bulk of funds go to the contractor.

Other organizations have studied various aspects of the sustainment problem. The Army examined costs of software sustainment and the work breakdown structure of sustainment as it is currently performed. Jack McGarry, author of the book Practical Software Measurement: Objective Information for Decision Makers, charted for the U.S. Army the tremendous variability in the demand for sustainment work, with the cost of the largest releases up to 10 times that of the smallest releases. The frequency of the largest releases appears to track to large-scale technology change, occurring about every four to seven years. In that period of time, the speed of processors increases by nearly 10 times, and the number of transistors on a chip also increases by 10 times. This finding suggests that technology change is an important variable in our systems dynamic model.

The research we've conducted thus far has revealed that our system dynamics model exhibits the expected and observed behavior of product sustainment. The model, however, has not yet been calibrated to apply to a real situation. We are actively seeking an opportunity to study a real sustainment program and to collaborate on calibrating the model. We believe that data collection will not be hard to implement and the data potentially can be used for other purposes by sustainers.

If you are interested in collaborating with SEI researchers on this initiative, please send an email to info@sei.cmu.edu, or leave contact information in the comments section below.

We welcome your feedback on our research and look forward to hearing if you'd like to help us achieve our project goals. Please leave feedback in the comments section below.

Additional Resources

To read An Examination of the U.S. Air Force's Aircraft Sustainment Needs in the Future and its Strategy to Meet Those Needs by the National Academies Press, please visit
https://www.nap.edu/catalog/13177/examination-of-the-us-air-forces-aircraft-sustainment-needs-in-the-future-and-its-strategy-to-meet-those-needs

To read the SEI technical report, Sustaining Software-Intensive Systems, please visit
https://resources.sei.cmu.edu/library/asset-view.cfm?assetID=7865

To see a presentation on Sustaining Software Intensive Systems -- A Conundrum please visit http://www.dtic.mil/ndia/2005systems/wednesday/lapham.pdf

To read Business Dynamics: Systems Thinking and Modeling for a Complex World by John Sterman, please visit https://www.amazon.com/Business-Dynamics-Systems-Thinking-Modeling/dp/007238915X

To view the proceedings of the 2012 PSMSC User Group, please visit
http://www.psmsc.com/UG2012/Workshops/w4-%20files.pdf

To read the Air Force Scientific Advisory Board report, Sustaining Aging Aircraft, please visit http://oai.dtic.mil/oai/oai?verb=getRecord&metadataPrefix=html&identifier=ADA562696