The modern software engineering practices of Agile and DevSecOps have revolutionized the practice of software engineering. These strategies have provided a foundation for producing working software products faster and more reliably than ever before. However, we found that most literature about Agile and DevSecOps focuses on the product and ignores the business mission and capability-delivery aspects of every software product under development. This gap is problematic because not every team in a software-oriented organization is directly involved with the development of the product and, in fact, the business mission and capability delivery aspects of DevSecOps are critical to the successful delivery of the product.

Expanding the view of DevSecOps beyond product development enables other teams to increase their capabilities and improve their processes. Though the primary use of Agile methodologies and DevSecOps principles may remain within the realm of software product development, “Agile methods are being used for complex system and hardware developments as well.” In this SEI Blog post, we share our experiences leveraging DevSecOps pipelines in support of teams focused on the capability delivery and business mission for their organizations.

The Bigger Picture

As outlined in Using Model-Based Systems Engineering (MBSE) to Assure a DevSecOps Pipeline is Sufficiently Secure and articulated in Figure 1 below, all software-oriented enterprises are driven by three concerns:

• business mission
• capability to deliver value
• products

The business mission captures stakeholder needs and channels the whole enterprise in meeting those needs. The business mission is owned by the organization’s leadership and is supported by various business functions, depending on the domain in which the enterprise operates. This part of the enterprise can answer the questions why and for whom the enterprise exists.

The capability to deliver value in a software-oriented enterprise covers the people, processes, and technology necessary to build, deploy, and operate the enterprise’s products. In general, this business concern consists of the software factory and product operational environments, however it does not consist of the products themselves.

Products, generically, are the units of value delivered by the organization. In a software-oriented enterprise, these products include the components, applications, services, and outputs that are delivered, deployed, and operated for use by the organization’s customers. These products utilize the capabilities delivered by the software factory and operational environments.

The enterprise provides the business case and requirements to each of the other concerns that are responsible for providing the capability to deliver value and the value itself. Both capability delivery and product development execute using different process steps to achieve their planned outcomes. They, however, need to synchronize with each other periodically to ensure that the software factory and operational environments remain capable of meeting the needs of the products under development.

Leveraging Pipelines to Implement Agile and DevSecOps Principles

We’ve established that not every team in a software-oriented organization is focused specifically on the engineering of the software product. We often encounter teams supporting the business mission or the capability to deliver value, but not the writing of code. These teams can still benefit from the development and use of continuous integration/continuous delivery (CI/CD) pipelines.

It’s almost impossible to discuss DevSecOps without talking about pipelines. Enablement of continuous integration/continuous delivery (CI/CD) pipelines is a key aspect of DevSecOps, and we consider it a best practice of the Agile methodology. As noted in an earlier post, a CI/CD pipeline provides for the automatic vetting, building, testing, packaging, and delivery of a change to a product in the desired destination.

The typical software-product-focused DevSecOps pipeline is depicted in Figure 2. As noted in the introduction to the DevSecOps Pipeline Independent Model, this pipeline seamlessly integrates “three traditional factions that sometimes have opposing interests: development; which values features; security, which values defensibility; and operations, which values stability.”

A simple implementation of the software-product-focused pipeline in Figure 2 can be described in the following steps:

1. A patch or new feature for the product is approved and assigned a priority for implementation.
2. A developer implements the selected patch and/or new feature for the product and submits the changes to a version-control system.
3. Automated build processes for the product are initiated.
4. If build processes are successful, automated tests are executed on the product, including verifying the developer-used proper secure-coding practices.
5. If all tests are successful, a release package is produced.
6. If packaging processes are successful, the release package is delivered manually to the production systems.
7. Package deliveries initiate automated installation or upgrade processes for the product on the production systems.
8. Over time, operations personnel implement and automate monitoring capabilities for the product.
9. A user observes the product’s functionality and requests a patch or a new feature in the product. This request is fed back to step 1.

As depicted in the infinity diagram, each of the steps in the pipeline integrates security controls appropriate for the product. However, this example pipeline may not be mapped readily to all situations. There are settings in which team processes would benefit from the use of a pipeline, but where the specific steps and procedures differ from this software-product-focused archetype.

In our various customer engagements, we have found many benefits in the application of a CI/CD pipeline, leveraging the principles and practices of Agile and DevSecOps to make informed decisions about process improvements.

Pipeline Applications in Atypical Settings

First Application: Software Development Environment

Our first example is firmly rooted in the Capability Delivery branch shown in Figure 1. In support of our government client, we worked on the deployment, operation, and ongoing maintenance of a development environment. This environment was designed to enable the product development teams to apply Agile and DevSecOps principles in the development of their software products. Tasks involved the set-up of physical servers, networking devices, a virtualization platform, a development platform, storage systems, and the management of endpoint system software.

The capability delivery team must balance three separate concerns: maintenance of policies and practices, maintenance of the infrastructure that hosts the product under development’s pipeline, and maintenance of the product’s pipeline. Each task might have a slightly different pipeline. All the competing-but-related concerns are maintained on their own cycles and timelines over the course of the development environment’s lifecycle. Though they all depend on one another, they also are separate entities.

In this environment, on the capability delivery team, our pipeline looked slightly different from the example above:

1. A fix or a new capability to be deployed in the environment is approved for implementation and assigned a priority for implementation.
2. The team evaluates the available technology and selects the required hardware and/or software to support the fix/new capability.
3. The team obtains the new hardware and/or software.
4. The team integrates the new hardware and/or software with the existing infrastructure (ideally in a test environment, not production!) and validates that it is working correctly.
5. The fix and/or new capability is made available to the end users.
6. Over time, operations personnel implement and automate monitoring capabilities for the product.
7. The team collects information from end users about how the fix/new capability is working and uses that information to inform the planning for the next set of fixes and features. This information is fed back into Step 1.

In this pipeline, many considerations not directly related to the development of the product may impact the product development teams. Step 1 of the pipeline must consider both improvements for the capability delivery team (including updates to low-level systems that keep the environment operating smoothly, like storage, networking, a domain name system (DNS), time synchronization, identity management, and configuration management) and developers’ services (such as a version-control system, container registry, and issue tracker). Developers care about whether the code repository is working correctly and whether they can access the build system. Both sets of priorities must be balanced during planning.

Second Application: Acquisitions Team

Our second example relates to the business mission of an organization. We worked with the acquisitions team to implement process improvements to their existing procedures for gathering approvals and documenting the acquisition process. We proposed and implemented a technical solution and then, as the team used the new technology, we used the iterative pipeline below to keep making improvements to the team’s processes.

At the beginning of the process, the team used a fully paper-based solution in which a folder of information was passed from person to person to review (and approve or reject) the submitted paperwork. Our first task was to gather requirements to understand the team’s current processes. Next, based on the team’s requirements, we set up the technical tools required to track acquisition requests. Then, we introduced the team to the new procedures that leveraged the technical tools. From there, we gathered feedback from the team and made necessary adjustments to make the system ready for users. The acquisitions team’s pipeline was also different from the standard software development pipeline.

1. Based on requirements and feedback from team members, change requests are submitted and then discussed to ensure that the changes will be beneficial overall.
2. The configuration of the technical tool is modified to reflect the selected change.
3. The team begins to use the updated tool configuration.
4. The team provides information on its experiences with the new tool configuration, both positive and negative, that is fed back into the planning stage where changes to the process are considered, prioritized, and implemented.

Over time, both requests and utilization began to become clear. This clarity stemmed from familiarity with the system and from the users' adoption of the Agile mindset of constantly improving the tool to make their lives easier. This familiarity was very encouraging and allowed fulfilling requests at a faster pace. Eventually, the system became more stable, and there were not any major changes via change requests. This situation allowed the acquisitions team to conduct an analysis and compare the rate of completion to the paper process.

Pipeline Comparison

In the two example pipelines we’ve presented, there are four common aspects over which the pipelines iterate:

1. Select a change.
2. Implement the change.
3. Observe the impact of the change.
4. Make an informed decision about the next change.

In our three examples, the main differences can be summarized briefly: the mechanisms for implementing the changes are different. There are different levels of automation able to be used in each case. There are different timelines for implementing and vetting each change. There are different costs to implementing each change; some changes require purchases, others require effort.

Agile Lessons Learned and Opportunities

Implementing the pipelines in these organizations was challenging, and we learned a lot through our efforts. First, because the DevSecOps literature presently available is focused on strategies for the product teams, it might not always be obvious that other types of teams can benefit from applying DevSecOps principles. However, it seems that this trend is starting to shift, and we are beginning to see more literature aimed at extending the application of DevSecOps and Agile into non-product spaces. The Platform Independent Model highlights this trend in its distinction between the Product Flow and System Flow. Practitioners are also discussing this trend with more frequency (https://www.usenix.org/conference/srecon22emea/presentation/looney). We think it’s important for DevSecOps practitioners to look for new opportunities to act as DevSecOps evangelists, who share the benefits of DevSecOps and Agile practices and principles whenever they have the opportunity to work with a new team.

Second, we’ve learned that sometimes the tasks not directly related to product development are not relevant to the product teams using the environments. Consequently, these tasks don’t get a high priority for implementation. When these items are under prioritized, the result is an environment that’s not as resilient and mature as it could be. At some point in the future, this will have a negative impact the product teams.

The solution to this challenge is to ensure that the ongoing maintenance of the capability delivery environment is allocated sufficient resources to attend to current and future issues. To allocate resources appropriately, we think it’s important for the capability delivery teams to engage regularly with the product teams and with the managers responsible for overseeing the business’s mission. In so doing, they should communicate the importance of the maintenance activities that impact the long-term health, security, and compliance of the capability.

Third, we don’t work in a vacuum, and we’ve learned that both the technical and cultural environments constantly change. Teams must assess and adapt to these changes to continue to operate at the highest level. For technical and non-technical teams alike, in both the capability-delivery and business-mission realms, documentation must be curated, new documents created as standard operating procedures change, and old documents are revised or removed as they become obsolete. Likewise, these teams must groom their task backlogs, removing those tasks that are no longer relevant and prioritizing tasks according to the current mission and objectives of the team.

Fourth, we’ve learned that it can be very tempting to be too agile, quickly abandoning a high-level plan in the face of unexpected technical issues. When problems are not handled properly, it is easy to both create technical debt and become consumed by the fire-fighting cadence. So, how do you avoid the technical debt? Begin by creating a high-level plan that has proper time and effort built in to allow each step in the plan to be properly implemented. This is not to say that the plan must be rigidly followed to completion (because that wouldn’t be following Agile principles) but to change the plan as needed and utilize it at every stage. Overcoming this challenge also requires good communication with the other related teams, especially the management team.

Finally, we have learned that these atypical applications of DevSecOps and Agile still encountered some of the same challenges faced by product teams. Namely, that change in an organization is hard, and Agile and DevSecOps practitioners should expect to meet some resistance and experience some pushback when advocating for the institution of Agile and DevSecOps practices. We have observed the following criticisms of the workflow changes:

• skepticism about the effectiveness of the new methodologies
• outright refusal to use new methodologies and tools
• the idea that ceremonies (i.e., meetings) are too much of a burden
• resistance to the idea of estimating how long some tasks will take
• hesitancy to join retrospective meetings where errors and mistakes are discussed

We certainly can’t provide solutions for every one of the cultural and personnel issues you may encounter, but we offer what we think are two keys to successfully navigating these challenges.

First, training and onboarding for the team can ease the assimilation process, ensuring that everyone on the team understands the overriding goals of the activities and ceremonies being performed. In some cases, healthy skepticism can lead to unique solutions. For additional insights into mitigating employee resistance, check out the blog post Six Remedies to Employee Resistance to DevOps.

Second, ensure that Agile ceremonies and DevSecOps practices and processes are strategically selected and executed. The selected ceremonies and processes should support the team’s goals and enable the team to consistently produce quality work. Selecting ceremonies and processes is a strategic activity that can produce benefits, such as a reduction in impediments to daily work and avoidance of technical debt. Execution of the selected ceremonies must be on point. Meetings viewed as too long or lacking in value are often met with disdain by participants. Effective ceremonies will encourage communication among the team, help the team maintain focus on a common goal, and foster the team’s understanding of how their work will achieve that goal. For more communication strategies, check out this webcast in which our colleagues offer tools to help organizations shift the culture to achieve DevSecOps.

The Benefits of Adaptation

In an effort to decrease the time and effort required to release software products, the DoD has long recommended the use of Agile methods in software engineering. Moreover, since 2009 the DoD has advocated for the use of Agile principles during the acquisition of IT systems. Our experience has shown that when used outside the scope of a traditional, product development setting, Agile and DevSecOps principles have the potential to improve the productivity of teams focused on business mission and capability deliveries.

The mindset, processes, and practices adopted from Agile and DevSecOps, including the use of iterative pipelines and the constant incorporation of feedback, can be applied in non-traditional ways by a variety of teams, with the ultimate goal of improving team processes. Inevitably, challenges will arise, but as we’ve discovered, there are valuable lessons to be learned and applied throughout the process, and we’re confident that the benefits of leveraging the tried-and-true Agile and DevSecOps principles outweigh the downsides.