A software product line is a collection of related products with shared software artifacts and engineering services that has been developed by a single organization intended to serve different missions and different customers. In industry, product lines provide both customer benefits (such as functionality, quality, and cost) and development organization benefits (such as time to market and price-margin). Moreover, these benefits last through multiple generations of products. This blog is the first in a series of three posts on sustaining product lines in terms of required decisions and potential benefits of proposed approaches. In this post, I identify the potential benefits of a product line and discuss contracting issues in the context of Department of Defense (DoD) programs.

The Challenge of Sustaining a Software Product Line in the DoD

Software product lines are unusual in programs partly because of the difficulties associated with the sustainment portion of the product lifecycle. Four factors highlight the difficulties in sustaining a product lifecycle in the DoD:

1. The cost of developing the initial product line is greater than for a single product.
2. Certain regulations and laws create a more complex contracting decision for sustainment.
3. Release planning is more difficult because of competing priorities among programs.
4. Supporting the product line over many years requires the ability of the maintainer of core components to recapture certain changes made by customers back into the core. The cost of this investment must be shared by the customers.

Decisions related to these factors make a product line a tough proposition in the world of DoD acquisition but not an impossibility. In this series of posts on sustaining of software product lines, therefore, I will explore

• required decisions and potential benefits of proposed approaches as well as potential benefits of a product line and discuss contracting issues (in this post).
• processes and goals for a governance structure associated to multiple programs and describe how to evaluate the success of the governance processes including the funding for the recapture problem (in the second post).
• problems and processes for release management (in the third post).

Product Line Benefits

The SEI has researched software product lines, both as case studies and by developing methods to design and build products. Our case studies demonstrate many significant benefits of software product lines, including (but not limited to) the following:

• large-scale productivity gains
• decreased time to market
• increased product quality
• decreased product risk
• increased market agility
• increased customer satisfaction
• more efficient use of human resources
• ability to maintain market presence
• ability to rapidly customize or configure the product
• ability to sustain unprecedented growth

For example, the Aegis Ballistic Missile Defense (BMD) System program is one of the most successful DoD program to operate a product line. Aegis began in 1970 under then Capt. Wayne Meyer. By the early 2000's it entailed 9 separate programs operating with a common architecture. In 2008, Lockheed-Martin developed a full product line. Aegis is now deployed on at least 33 ships, several ground bases and by certain U.S. allies. A good description of the history, product structure, governance and benefits of AEGIS appears in this paper by Gregg et. al.

A significant and successful commercial product line has been developed by Rolls Royce for its jet engines including both hardware elements (e.g., the turbine design) and software elements (fuel control) of the Rolls Royce family. Another example of a successful commercial product line is the Android Open Source Project (AOSP), which provides millions of lines of source code based on a common architecture that is used to create custom variants of the Android stack on hundreds of different hardware devices provided by dozens of OEMs.

Recently, the SEI has seen multiple examples of product lines being used to supply similar yet different configurations and capabilities across DoD Services to meet the unique needs of communication and ground control systems. Capabilities like mission planning and user interfaces are separated out to enable quick and efficient changes. One product line example that we have encountered with three operational systems shows that the quality of the core software is now exceptionally high, exceeding Level 2 benchmark of 0.1 defects per KLOC suggested by the Coverity 2017 Scan Report. Since high quality software reduces rework costs and improves staff efficiency, product teams can spend more time on new capability and improved product performance. Less time is required to add new products to the product line, as well.

Defining Software Sustainment

Software sustainment involves coordinating the processes, procedures, people, information, and databases required to support, maintain, and operate software-reliant aspects of DoD systems. Federal regulations for sustainment classify work into two budget categories: a one-year budget for operations maintenance (O&M) and a two to three-year budget for research and development (RDT&E or R&D for short).

Hardware O&M covers both periodic maintenance, spare parts, and repair. These maintenance activities are typically supported by trained technicians. Budgeting is based on the size and utilization of the inventory and is fairly predictable. Engineering support is required when changes to mission, supply chain, or other interfacing systems dictate. Occasionally, updates requiring engineering support are sufficiently complex to require a separate R&D budget to upgrade the product.

Software does not exhibit wear so there is no "changing the oil." Every change to software has some direct effect on the product design and typically involves multiple phases of the development lifecycle: specification, design, compilation, and testing. Thus, every change to software involves engineering staff. While some changes may fit within a yearly funding model, many software programs bundle changes into longer projects. The full development lifecycle for an enhancement release may take five years or more. The first two types - corrective and adaptive changes - belong mostly in the O&M budget. The last two types belong mostly in the R&D budget.

• Corrective:
  fixing bugs or defects in the software
• Adaptive:
  changes that must be made to recover lost functionality and performance caused by changes made in other systems
• Perfective:
  changes to the software to improve performance or add functionality
• Preventive:
  changes to software intended to prevent cyber threats or improve maintainability and reduce the cost of ownership

Since the approval process for R&D funding takes longer and only occurs every two years, perfective and preventive changes are often delayed. The term "technical debt" is often used to describe this loss of capability. Eventually a major upgrade is required. An interesting Army study on sustainment has shown that such a major upgrade occurs every 7-to- 10 releases at a cost 10-to- 20 times the normal O&M budget.

Contracting Decisions

In traditional DoD sustainment, support may be conducted in the field, at a depot or by specific contract. Software support requires relatively little field support once installation is complete. Since creating a new contract is a time-consuming and expensive proposition, depot level support is highly desirable. Another government regulation (Title 10 Section 2466 of US code) requires that 50% of depot level work is assigned to DoD personnel. At most 50% of the work can go to a contractor. This "50/50 rule," dividing software sustainment responsibilities between the contractor and organic support makes for a complicated set of trades between time-to-market, quality of product and cost of support. In particular, the question is "who should perform support for the core of the software product line." Since the core components are common across the programs, assigning the core to a contractor where costs can be shared across multiple programs should make it easier to meet the 50/50 rule.

The next question is the level of support required to maintain the core components. A benchmark for nominal support of software suggests that a single engineer can maintain 50-100 KLOC. In one product line, the shared core is about 1 MLOC. So we can estimate cost of maintaining these core products between 10-to-20 staff.

The core maintainer would require an additional 1-2 persons per program supported to maintain the tight relationship required between program and core. The core maintainer would therefore need 15-25 staff to support core plus 5 programs. Dividing this cost among 5 programs suggests that each product program pays a fee to the core maintainer of 3-5 staff. Even though the cost savings are significant, the contractual relationship will be complicated since responsibility for both O&M and R&D budgets has to be allocated across both core and programs. Additional funding for both program and core would be required with significant enhancement and technology upgrades.

The product line approach suggests there is value to a formal public-private partnership. A potential model could use DoD personnel (called "organic support") to focus on high-value mission related changes that are typically smaller and require faster time to market. The contractor can then focus on driving quality and product stability to the core elements of the product line. In addition, the contractor has the role to capture reusable elements from the organic team into the core for common use across multiple programs. The public-private partnership (PPP) should work for many product lines. The cost-benefit of the PPP is significant provided that the programs can establish an appropriate working relationship with an emphasis on quality and functionality.

Whatever organization assumes responsibility for core components, it must have a measure of independence from the development of the specific products. Moreover, it is important that the core maintainer is able to charge a fee to each of the specific products in order to continue developing and refining the core. The core maintainer must therefore be somewhat independent of the political winds affecting any specific product.

If a contractor assumes the responsibility for core components, it must be prepared to defend the practice when there are new contracts that could potentially use core components. The software product line should provide a sufficient edge in competing the contract but a longer-term, sole-source contract might save the government a great deal of time and money. The various program offices might take a position supporting the sole source contract if they found sufficient benefit from the existing product line approach.

Wrapping Up and Looking Ahead

Product lines enable the longevity of software baselines. Over the years, performance goals will change depending on the needs of the various systems being deployed. Sometimes there is a need to maintain and improve system availability. At other times, there is a need to deploy updates quickly with the minimum set of new features working correctly. Changing performance goals affect changes in the basic design and always require a set of engineering decisions. Balancing the timing and priority of the entire list of sustainment changes is a challenge for all systems. The ultimate goal of this blog series is to identify desired outcomes and data sources, thereby helping acquirers and contractors to make better sustainment decisions for DoD systems.

Additional Resources

Read the series by Mike Phillips on Efficient and Effective Software Sustainment.

View the podcast Software Sustainment and Product Lines by Mike Phillips and Harry Levinson.