Although software is increasingly important to the success of government programs, there is often little consideration given to its impact on early key program decisions. The Carnegie Mellon University Software Engineering Institute (SEI) is conducting a multi-phase research initiative aimed at answering the question: is the probability of a program's success improved through deliberately producing a program acquisition strategy and software architecture that are mutually constrained and aligned?

Moreover, can we develop a method that helps government program offices produce such alignment? This blog post, the third in a series on this multi-year research, describes our approach to determining how acquisition quality attributes can be expressed and used to facilitate alignment among the software architecture and acquisition strategy.

In the first post in this series, we identified specific instances where misalignment between software architecture and acquisition strategy resulted in program delays and cost overruns--and, in some cases, program cancelation. In the second post, we outlined seven patterns of failing behavior, or "anti-patterns," that were major contributors to this misalignment. In addition, we presented a model that links key program entities and their relationships that could avoid those anti-patterns. Finally, we observed that alignment among software architecture and acquisition strategy does not occur naturally. Of particular note in this early research, we postulated that acquisition quality attributes reflecting the program's business goals can be used to judge the effectiveness of an acquisition strategy--analogous to software quality attributes reflecting the mission goals that are used to judge the effectiveness of a software architecture.

In this latest effort, a team of researchers that, in addition to myself includes Cecilia Albert, Patrick Place, and David Carney, have focused on determining how to express, elicit, and analyze acquisition quality attributes for their utility in surfacing potential incompatibilities between a software architecture and an acquisition strategy.

Our Research Approach

We patterned our approach to defining acquisition quality attributes on the original SEI research on software quality attributes. We began by forming a starting starter set of potential acquisition quality attributes. We compiled an unordered list of roughly 30 qualities derived from a review of government acquisition strategy guidance and discussion with acquisition professionals, colleagues, and several brainstorming sessions within our team. Examples of these initial acquisition quality attributes are criteria by which acquisition strategies are judged

• affordability
• achievability
• effectiveness
• flexibility

Next, we defined an approach for expressing program-specific acquisition quality attribute scenarios using the SEI's earlier work in software architecture where stakeholders are encouraged to create small "stories" that specify some event (the stimulus) that occurs in a particular part of the lifecycle (the environment) and a desired behavior (the response).

For example, in software architecture, a quality attribute might be expressed using the following three-part scenario:

• stimulus - an internal component fails
• environment - during normal operation
• response - the system recognizes an internal component failure and has strategies to compensate for the fault

In the acquisition domain, a similar example might be as follows:

• stimulus - an unexpected budget cut
• environment - for a multi-segment system
• response - the program is able to move work between major segments to speed up or slow down separate segments within the available funding

Next, we elicited acquisition quality attribute scenarios from former program management office personnel and members of Independent Technical Assessment teams (informally called Red Teams) to capture 55 acquisition quality attribute scenarios covering 23 government programs. We needed these scenarios to validate that qualities related to an acquisition strategy can be expressed in a meaningful way through these scenarios. In addition, we were able to show that a tight link exists between the acquisition quality attribute scenario and some element of the of the acquisition strategy. These scenarios are detailed in a technical report, Results in Relating Quality Attributes to Acquisition Strategies.

The following is an example of the scenarios we constructed:

• stimulus - a new need arises when we want to react quickly
• environment - where there are only a limited number of contractors able to do the work
• response - the program can satisfy the need by adding the work to an existing contract
• potential acquisition tactic - Award IDIQ contracts to multiple (perhaps eight or so) vendors and issue task orders in a round-robin fashion

We built and validated a prototype workshop adapted from the software Quality Attribute Workshop (QAW). The major changes we made to a conventional QAW were to place more emphasis on business presentation and replace the architecture presentation with one on the program's acquisition strategy plans, thus we termed our revised approach the Acquisition Quality Attribute Workshop (AQAW). We then prototyped our AQAW using a real program but substituting SEI staff for various program stakeholders. The prototype AQAW successfully generated twenty acquisition quality attribute scenarios. While only a single instance, the prototype successfully demonstrated that an AQAW is a plausible approach for capturing acquisition quality attribute scenarios. Finally, we analyzed all captured acquisition quality attribute scenarios for trends, implications, and potential incompatibilities. Some of our findings are described below.

Different scenarios result in different acquisition strategies. For an acquisition quality attribute scenario to influence the acquisition strategy, there must be some element of the scenario that leads the program office to choose a strategy. For instance, we found a number of acquisition quality attribute scenarios relating to new technology and the issues that arise if the chosen innovative technology fails to deliver on its promises:

1. A new technology the program office expects to use is found to be unsuitable where schedule is of prime importance; the program office switches to an alternative that is also currently under development and is evaluated to be suitable.
2. A new technology the program office expects to use is found to be unsuitable where costs must be kept as low as possible; the program office instructs the contractor to restart but using an alternative technology.

In the case of these scenarios, the stimulus is the same but the environment changes. In the first scenario, schedule is more important than cost. The second scenario reverses their relative importance. In the first scenario, an acquisition strategy starting multiple developments simultaneously with a requirement for some kind of decision between the alternatives would be appropriate. In the second scenario, a strategy starting a single development contract and continuing to use it while switching to a more feasible technology might be more appropriate.

Although this is a simple example, it demonstrates that different acquisition quality attribute scenarios can lead to different acquisition strategies. This finding strengthens our contention that our use of acquisition quality attributes and acquisition quality attribute scenarios is analogous to the use of software quality attribute and software quality attribute scenarios and that we may continue to rely on methods and mechanisms developed for that purpose to assist with the creation of sound acquisition strategies.

Incompatible Acquisition Scenarios. Conflicts between scenarios are not always obvious, and may be quite subtle without some analysis. For example, organization ABC is using a large, complex legacy system deployed in multiple operational locations, where each location installed its own local variant of the system. Over time, these variants have diverged in response to differing requirements of the local users. To accommodate mission changes, it has become important to share data across these locations in a more integrated way. A new program was initiated to acquire one replacement capability that would support all of the differing needs across the multiple fielded locations. The program decided to implement an incremental approach to replacing the legacy system so they could respond to budgetary constraints and uncertainties.

For the example described above, the following scenario reflects the expectation of one set of influential stakeholders who advocated the use of a commercial off-the-shelf (COTS) product that had been successfully used at one of the operational installations:

• stimulus - There is a desire to replace a complex component of a large legacy system with a COTS package
• environment - Within an established enterprise architecture with many local variations implemented that are largely different from each other
• response - The program runs a competition to evaluate COTS packages for an enterprise-wide solution.

A second set of stakeholders, reflecting operational users, is counting on the new system to quickly address their current needs. These needs vary among the current fielded locations. During the time it takes the program to define an agreed upon set of requirements for each increment, the user representatives from the various fielded location change their requirements. This leads to a second acquisition scenario:

• stimulus - Requirements for the next release keep changing
• environment - For a program with a fixed budget that must be carefully managed
• response - The program accepts the new requirements

Implied in these two scenarios is a third set of stakeholders: the enterprise system engineers, who are advocating the implementation of an enterprise architecture that extends across all of the local fielded implementations. This enterprise architecture could be incompatible with both of the above scenarios: each COTS product, by definition, is built to an architecture and a set of requirements that organization ABC has no control over. Moreover, the demands for local fielded implementations compete with architectural changes within a constrained budget.

Unfortunately, the two scenarios are potentially incompatible with respect to designing the acquisition strategy. The first scenario describing the implementation of a common COTS product across all locations could provide sizeable value in terms of moving to one capability that is used across all fielded locations, but it may not meet what the current users described in the second scenario consider urgent needs--and both of these may conflict with the move to an enterprise architecture.

Looking Ahead

We were pleased to demonstrate a critical link between acquisition quality attributes and an acquisition strategy. We also identified potential recommendations for how an acquisition quality attributes could be expressed in program specific three-part scenarios. In the next phase of this work--already under way--we are working to develop an alignment method that collects and analyzes acquisition and software quality attributes in a way that enables deconfliction and prioritization. We believe that acquisition strategies and software architectures built from this consistent set of quality attributes would be aligned and mutually constraining.

Additional Resources

To read the SEI technical note, Results in Relating Quality Attributes to Acquisition Strategies, please visit,
https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=78312

To read the SEI technical report, Quality Attribute Workshops, please visit
https://resources.sei.cmu.edu/library/asset-view.cfm?assetID=6687

To read the SEI technical report, Isolating Patterns of Failure in Department of Defense Acquisition, please visit https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=53252