This post examines problems that arise from a shared DevSecOps platform. Because a DevSecOps platform and tool pipeline is too complex and expensive to create and manage separately for each program, the platform often needs to be a shared capability. This situation creates dependencies and cooperation issues.

These problems are examples of an acquisition archetype, which is how we refer to a pattern of organizational system behaviors that have been seen during the SEI’s experiences in conducting invited independent technical assessments (ITAs) on technical and programmatic aspects of the DoD acquisition programs. In these ITAs, program management office (PMO) staff, contractor staff, users, and other external stakeholder organizations provide open and candid responses under the condition of anonymity that provide the SEI team insight into what is truly happening in a program. These insights suggest that similar, recurring problems in software acquisition and development—archetypes—arise across separate and seemingly dissimilar programs.

A previous SEI Blog post examined an archetype of clinging to the old ways. In this post, I discuss the recurring problem of cross-program dependencies. I describe the behavior in the context of a real-world scenario and provide recommendations on recovering from and preventing future occurrences of this problem.

About Acquisition Archetypes

Our use of the phrase, “acquisition archetypes” is based on the more general notion of system archetypes and is meant to describe recurring patterns of failure observed in acquisition programs to raise awareness, along with providing approaches to mitigate or avoid those adverse patterns. The incentives that drive these patterns are similar across programs and tend to drive similar behaviors.

Cross-Program Dependencies

Sometimes an organization may need to build a new common infrastructure capability. For instance, an organization might deploy a DevSecOps platform and tool pipeline (e.g., compilers, code scanners, containers, and orchestration) that is too complex and expensive to create and manage separately for each program or project. These programs or projects might be reluctant to accept an external dependency on the infrastructure program offering a common infrastructure capability, leading to internal tension. If the common infrastructure has issues such as poor performance, difficulty of integration, inability to fully perform its function, or unavailability during the required timeframe, the projects providing and supporting that capability are likely to become disenchanted or reluctant to continue to support the infrastructure, and may criticize or even undermine it. For example, existing programs migrating to use the infrastructure might be familiar with using a particular model-based systems engineering (MBSE) tool or a code scanner that implements a specific set of scanning rules. Making the change from using the tool they’re familiar with to using an entirely different tool will have both up-front costs in terms of modifications to the tools and the system, and longer-term costs as users must learn new ways to accomplish the same effect.

Projects using DevSecOps infrastructure will often need to make significant changes to their portions of the capability to accommodate the new infrastructure (e.g., changed interfaces, additional functionality, or architectural differences). Supporting the new infrastructure will make their own work more challenging, require additional effort and resources, adversely affect their existing systems, and require rework of aspects of those systems. Consequently, these projects have little incentive to fully support the new system. Rather than being a win-win across the organization, the common DevSecOps infrastructure may become primarily a win for headquarters at the expense of the other projects.

Report from the Field

The way a program is established affects the ability of multiple, semi-independent organizations to cooperate to achieve a common goal (Figure 1). In the course of supporting acquisition programs, the SEI often encounters and must help address these types of organizational issues. In one such conversation a program leader said, “Some programs get concerned when they have dependencies on other programs. It’s a problem when different groups control different services, and you don’t have it all under your control.... The infrastructure program asks us for stuff, and sometimes there’s funding, and sometimes there isn’t.” Another leader stated that, in delivering capabilities, “Different organizations are in charge, funded separately, with different budgets, and so they’re required to deliver toward requirements that don’t take into account things they might want.”

In one case, “[a] PMO wasn’t prepared for the inevitable bow wave of new work that was coming their way. They didn’t like being told what to do [by a higher authority akin to a program executive office or PEO]. That created some contention.” This situation sometimes devolved into finger pointing, rather than producing results: “The different organizations involved all have to work together to share requirements and make decisions—but no one owns it, so they blame each other.” If the directing authority had been able to offer schedule relief and/or funding for the additional work, it might not have been viewed by the PMO as essentially an “unfunded mandate.”

In this case there was a misalignment of goals that each different organization was trying to achieve. One official observed, “The motivation at our program office is to meet cost and schedule performance, while the infrastructure program is about capability delivery and quality. Subsequently, the relationship mismatch distracts from efficiency.”

Analysis

Organizational tensions can occur due to the reluctance of programs to accept an external dependency on another program that would help to provide a common infrastructure capability. The causal loop diagram (CLD) in Figure 1 represents multiple interacting programs and shows that the way one program is established can affect its ability to cooperate with other programs as they all reach toward a common goal. The leftmost loop (in green) shows that the less able the “consuming” program is to achieve their goals by themselves, the more they need the shared infrastructure. The rightmost loop (in gold) shows that when a “producer” organization is tasked to provide shared infrastructure for multiple programs but is unable to meet all of the “consumer” organizations’ expectations, the consumers may become dissatisfied and decide to construct their own private or custom versions of the infrastructure instead. However, the middle loop (in red) shows how doing so generally undermines the desired degree of interoperability the shared infrastructure was intended to enable. Constructing multiple, less-interoperable, private versions of the infrastructure costs significantly more than a single shared version, using up funding that could have been spent to build the shared infrastructure. These private versions are the result of desiring an immediate benefit (removing the dependency) that will cost everyone else—but if everyone does the same thing, everyone ends up worse off due to the additional development costs, non-standard systems, and schedule spent in development and rework of the results. This is a balancing loop, which oscillates around an equilibrium value as support for the infrastructure grows and then wanes. Note that the static model described by this CLD doesn’t predict how this dynamic will play out in all circumstances but does describe how it often ends with consumer programs opting out of the shared infrastructure arrangement if they can.

Solutions and Mitigations

A public good is an economics term for a service that is made available to all members of a community where use by one member does not preclude its use by others. For example, our national defense itself is a public good for all citizens. The dynamic of producing a public good in human organizations has been researched extensively and has a large set of standard solutions. The development and provision of common infrastructure, such as a DevSecOps platform, is a type of public good that is subject to cooperation problems from cross-program dependencies.

In dealing with cooperation problems, there are three classes of solutions: motivational, strategic, and structural. These are broadly characterized as follows:

• Structural: Reframe the problem and rules so that people must behave more cooperatively because there is formal authority behind, and enforcement of, the rules (e.g., penalties, laws).
• Strategic: Give people a rational and self-interested reason (i.e., incentive) to behave more cooperatively.
• Motivational: Make people feel differently so that they want to behave more cooperatively.

The cross-program dependencies dynamic can be managed by leadership that can recognize these dependencies as they arise and take steps to mitigate them. However, in this scenario the leadership would need to be at or above the PEO level, and the anticipated adverse ramifications of the issue would need to be raised to their attention by one or more of the programs involved. Hierarchical, authority-based organizations such as the military take this approach, although usually after discussion with the affected parties. It is a structural solution, often referred to as “regulation by an authority,” but it requires having an authority to impose the rules, may need enforcement of compliance, and may inspire resistance from those it is imposed upon.

If a common infrastructure program has overarching authority over the projects providing supporting capabilities, it can address many of the issues noted above. However, the way such authority could be granted would vary substantially throughout the DoD, and in some circumstances may not always be possible. When it is possible, it may also happen that such authority is overused, even when the infrastructure program has the best of intentions in exercising it. The authority could override the desires or needs of the participating projects; for example, it might force participating programs to implement unfunded or even undesirable mandates.

Wherever possible, the authority of the common infrastructure program should be exercised in win-win arrangements that try to provide value in both directions, to both parties. Win-win relationships can involve providing the supporting projects what they want (e.g., funding or resources), fixing issues they might have by providing organizational expertise, offering specialized training or support that they need, and/or finding ways to burnish their reputation.

The second way to address cross-program dependencies is through strategic approaches, such as setting up a meaningful incentive that rewards cooperation to drive successful joint end-to-end outcomes for users. These approaches are weaker than structural approaches, but can be used to augment them and include:

• establishing cross-fertilization/cross-functional teams (exchanging people to break down barriers and encourage cooperation)
• creating more interdependencies (encouraging people to work together out of necessity).

The third way to address cross-functional dependencies is through less formal motivational approaches. These approaches try to mitigate loss of trust and cooperation among the different projects supporting the common infrastructure by using activities that help maintain or rebuild trust. While weaker than either of the other two, these can also be used to augment structural and strategic approaches. Possible motivational approaches for addressing the behavior could include:

• Awareness: Increase the awareness of the problem and communicate the importance of everyone making a difference to resolve it.
• Proof of quality: Provide compelling proof that the product or service will work as advertised before asking organizations to support it or help pay for it.
• Setting expectations: Encourage voluntary cooperation in settings in which people are more likely to be public-minded due to history and tradition (e.g., Peace Corps or War Bonds).

The Outlook for Cross-Functional Dependencies

In this post, I’ve investigated one recurring program behavior related to the introduction of DevSecOps: cross-functional dependencies. DevSecOps is a powerful approach that raises new considerations around cross-functional dependencies. The complexities of DevSecOps can require programs to make infrastructure changes that can have significant downstream effects for other programs and projects. By developing mutually beneficial solutions, the authority of the common infrastructure program can encourage cooperation and better behavior.