Identifying, estimating, and containing the cost of software is critical to the effective deployment of government systems. Cost estimation has been cited by the Government Accountability Office (GAO) as one of the primary reasons for DoD programs' cost overruns. Planners typically estimate costs via modeling and simulation tools, such as the Constructive Cost Model (COCOMO II). While COCOMO II is primarily used to estimate development costs, the costs of operating, maintaining, and sustaining software-intensive systems are often harder to estimate and predict than development costs--and more significant. It is often these operation and maintenance (O&M) costs that are most responsible for problematic cost overruns. This post describes SEI efforts to control cost overruns and reduce costs over the system's active lifetime by including open systems architecture considerations in the O&M costs early in the software-development lifecycle.

Software Architecture

One way to predict software costs earlier in the lifecycle is to focus on software architecture. The architecture of a software system largely determines the ease and complexity of ongoing O&M. As time passes, functionality and maintenance requirements change, new components are added to the system, and the system adapts to new missions for which it may not initially have been intended. An examination of the software architecture of the system provides insights into the cost of maintaining long-lived systems over time.

Open Systems Architecture

To deliver enhanced capability at lower cost, the DoD is moving away from independent, one-time solutions and embracing open systems architecture approaches. Open systems architectures integrate business and technical practices to create software-reliant systems with interoperable and reusable components. The promise of open systems architectures is their potential to reduce the costs of O&M later in the lifecycle. In particular, key benefits of open systems architectures include

• improved sustainability properties
• ability to upgrade more quickly and at lower cost
• ability to add new functionality to single software components simply, quickly, and for less money
• ability for the system to exchange data consistently among components
• ability for a third-party provider to develop an application or replace a defined subsystem to work with the system without any assistance from the system provider
• ability to update system functionality remotely, dramatically adding to the flexibility of the system in operation

It is clear from this description how the openness of the software architecture in a software-reliant system can reduce O&M costs. Our goal at the SEI is to provide objective criteria for measuring the openness of a software architecture and use that information to estimate software development and maintenance costs more accurately. It's our experience that objective measures of openness will help managers, program executive offices, reviewers, and tiger teams predict and reduce O&M costs in long-lived software-reliant systems. To this end, the SEI has developed the toolset depicted in Figure 1 that provides

• a configurable open systems architecture assessment with quantitative ratings for a software architecture
• a software cost-estimation tool across the full lifecycle that accounts for open systems architecture ratings

Figure 1: The SEI Cost-Estimation Toolset

This SEI work incorporates previous work in the areas of software architecture evaluation, software cost modeling, and assessment of open systems architectures. Figure 1 shows how the toolset enables a program to perform an open systems architecture assessment on a selected software architecture and then to provide software cost-estimation inputs, including assessment ratings, to a COCOMO II-based software cost-estimation tool (MS Excel) for the selected software architecture. A widget in a popular statistical analysis language called R combines the COCOMO II parameters with the open systems architecture ratings. These results can then be applied to compare predicted software lifecycle cost estimates among multiple software-architecture alternatives (e.g., legacy, modified legacy, modernized and legacy, completely modernized) early in the system lifecycle.

How to Assess the Openness of an Architecture

The SEI cost-estimation toolset calculates weighted ratings for four attributes common to all open-systems-architecture assessment methods and related materials, such as the Defense Acquisition University (DAU) Open Architecture Assessment Tool (OAAT), Open Group Future Airborne Capability Environment (FACE) Consortium, and the Air Force Air Worthiness Study. These and other open systems architecture assessment methods weighted the following attributes heavily:

• modularity. System architecture key components are encapsulated, cohesive, self-contained, and loosely coupled.
• interface standards. A widely available document exists that specifies interfaces including services provided/required, protocols, message and data formats, etc.
• layering and tiers. A software abstraction provides separation from other software packages and technology.
• open and accessible standards. Key interfaces are based on open and accessible standards that are widely used, consensus based, published, and maintained by recognized communities of interest.

We apply this methodology to the software architecture for each unit of assessment (UoA) in a software-reliant system. For example, a DoD combat aircraft can be decomposed into separate UoAs: radar system UoA, weapons system UoA, displays and control UoA, mission planning UoA, each of which we assess individually. Within a UoA, we measure the openness of the software components (based on the key attributes above) of the high-level software architecture that compose the UoA and feed that data into an updated COCOMO II model.

Our methodology follows the COCOMO II model by predicting costs on a component-by-component basis, applying basic checklists for modularity, interface standards, layering, and open/accessible standards. Although the SEI provides guidance when working with programs, these attributes can be weighted according to which fields a program office deems important in the context of its program. Program managers can also add other attributes.

Cost Estimation Based on System Openness

As shown in Figure 1, the SEI cost-estimation toolset takes as input the results of a software openness checklist and converts it into an estimate of software development and maintenance costs. This output includes a modified version of a configurable open systems architecture checklist tool and an implementation of the COCOMO II estimation equations. The parameters of the cost model are adjusted by the quantified ratings of openness that are computed by the checklist tool.

The toolset permits reasoning about the effects of architectural modifications on system costs by

• comparing a baseline system with one modified to support openness
• comparing different architectural approaches to achieve the same architectural quality
• investigating the tradeoffs of openness versus other quality attributes, such as security or dependability.

As demonstrated in the example below, the SEI cost-estimation toolset can produce graphical representations of how estimated costs change over time. Our goal is to produce evidence that including software architecture in a system's early tradeoff decisions can yield a better system architecture that has less risk and less cost (due to avoided rework) over the system's lifecycle.

Quantifying Cost Savings from Transition to Open Systems Architecture

Our toolset enables us to compare software cost per year for two alternative trajectories in typical DoD acquisition programs:

1. Keeping the legacy architecture, with its high maintenance cost. While this trajectory involves no development costs, it can incur high maintenance costs based on the original architecture's lack of openness.
2. Continuing to use the legacy architecture simultaneously, while developing a new, more open architecture that will have lower maintenance costs and switching over to the new system after several years. This trajectory increases the development costs of rearchitecting the software for the first few years to the ongoing maintenance cost for the old architecture during this same time. After the new architecture is in place, however, this second trajectory includes only the lower maintenance cost for the new, open-architecture software.

To show the potential utility of our toolset, we use a hypothetical DoD acquisition program as an example, comparing two potential architecture trajectories described above. By applying the SEI cost-estimation toolset we can show that the second trajectory will ultimately yield significant cost savings based on the architecture-evaluation criteria that produce the ratings.

Figure 2: Hypothetical Armored Vehicle Example

In Figure 2, Trajectory 1 is represented by the red line that goes up--the original software architecture that is maintained for 15 years. The blue line, Trajectory 2, shows three years of increased costs that represent maintaining the old software while simultaneously developing new software based on open architecture. At Year Three, maintenance costs flatten. When we switch over to the new, better architected system in Year Three, the graph shows that the program would begin recovering its rearchitecting investment in Year Nine. By the end of the time period shown in Figure 2, therefore, the program would save $75 million.

Wrapping Up and Looking Ahead

Ultimately, our work on the SEI cost-estimation toolset aims to equip acquisition program staff with as much insight as possible into the implications on the software-development effort of early system and software architecture decisions. Acquisition programs are sometimes unaware that early program decisions have specific software architectural implications. Our cost-estimation toolset clarifies these early decisions and quantifies them to the point where their downstream effects on system lifecycle costs can be demonstrated conclusively.

The work described in this blog post has the potential to remove uncertainty about cost as a barrier to adopting open systems architecture methods, platforms, and tools. Our toolset can be used to evaluate candidate architectures for systems before they are developed in terms of anticipated development and O&M costs. We invite programs that would like to learn more about this work to contact us.

Additional Resources

Read other SEI blog posts about open systems architectures.

Read other SEI blog posts about software architecture.

Read other SEI blog posts about software cost estimates.