Implementing DevSecOps can improve multiple aspects of the effectiveness of a software organization and the quality of the software for which it is responsible. Implementation of DevSecOps is a complex process, however, and the way a program evaluates progress in its DevSecOps implementation is important. We propose here a frame of reference for DevSecOps maturity, enabling organizations to focus on outcomes – value delivered – without excessive focus on compliance.

The Department of Defense’s (DoD) DevSecOps Documentation Set emphasizes program activities that speed delivery, tighten security, and improve collaboration across the software development lifecycle. But without a deep understanding of the interdependencies between the roles and activities within a DevSecOps ecosystem, less beneficial sub-activities could be optimized at the expense of others that might be more beneficial, resulting in waste. Effective DevSecOps ecosystems must be based on objective observations and data that account for the journey a software program undergoes as it implements and improves its DevSecOps capabilities.

Evaluating DevSecOps implementation activities against a set of characteristics, attributes, indicators, and patterns in not sufficient. It must be done within the context of value delivered. Therefore, in this blog post, we first define value in a DevSecOps context. Next, we describe how the DevSecOps Platform Independent Model (PIM) provides an authoritative reference model for evaluating an organization’s DevSecOps capability maturity. Finally, we provide a benchmark example of a DevSecOps capability profile.

What Is a Maturity Model?

A maturity model is an identified set of characteristics, attributes, indicators, and patterns that represent progression and achievement in a particular domain or discipline. It allows an organization, such as a software factory, to assess its practices, processes, and methods against a clearly defined benchmark. A scale of capability maturity levels can be established as an evolutionary scale that defines measurable distinctions from one level of capability to another. Maturity models can be used to:

• Determine an organization’s current level of capability and then apply these methods over time to drive improvements
• Determine how well an organization is performing relative to others by examining the capabilities of peer organizations

It is important for organizations to perform evaluations with value in mind, as the value proposition is required to define the scope and perspective of a DevSecOps capability assessment.

Understanding Value within a DevSecOps Perspective

The practice of DevSecOps equips people in an organization with the tools and processes necessary to deliver value in the form of working and secure software to users quickly and reliably. It requires that the organization adopt a culture and organizational structure aligned with Agile and Lean principles.

Value is fundamentally measured by mission impact—how and how much do the software products that the team delivers impact the capability and effectiveness of performance of a mission set? A consequence of this definition is that value cannot be realized until the product is not just delivered and deployed but also used to complete missions. DevSecOps is therefore structured not to stop at delivery or deployment, but rather to continue through operations – and to loop back to development so that the software benefits from feedback from real users on real missions. See Figure 1.

How Value Drives Scope

DevSecOps is not something you buy; it is something that an organization (or enterprise) is. It embodies the guiding principles of Agile and Lean software development. DevSecOps combines organization context and culture with practices and tools:

• Business Mission: captures stakeholder needs and channels the whole program in meeting those needs. It answers the questions Why and For Whom the enterprise exists.
• Capability to Deliver Value: covers the people, processes, and technology necessary to build, deploy, and operate the enterprise’s products.
• Products: the units of value delivered by the program. Products utilize the capabilities delivered by the software factory and operational environments.

All these aspects must be brought together into a single organization, ideally under a single DevSecOps product owner, with the focus on delivering valuable products to the user community. It may not be possible for the DevSecOps product owner to own all teams and processes necessary to deliver value; however, it is imperative that they own the full end-to-end process of delivering that value. Lean practices can help enable a DevSecOps product owner to more readily identify wasteful, redundant, and otherwise unnecessary tasks in the current set of processes and optimize those that remain. Even if they cannot fully control external stakeholders, they are best positioned to mitigate the impacts of inefficiency in those processes by optimizing and realigning the processes that they do control. For example, an organization must follow an external approval process before the recipient can install and operate a delivered application. If this process is costly or takes a week or more, and the product owner cannot currently optimize that time frame, the product owner could instead decide to reduce the frequency of delivery and lengthen the development cycle so that delivered software has a chance to get through that approval process, get installed, and get feedback to the development teams before the next scheduled delivery. This alignment of frequency of delivery to operational acceptance rate is crucial to optimize flow, but only a stakeholder with insight into the entire process can recognize this and adapt.

How Capability Evolves

What DevSecOps brings to the table is the automation to improve the agility and quality of software in a way that is repeatable, predictable, reliable, timely, and secure. As shown in Figure 3 below, this is an iterative process. DevSecOps incorporates automation to streamline processes, perform repeated tasks, complete tasks faster, and reduce human error. Automation, however, first requires a well-defined set of processes that the teams can consistently and reliably execute and that have demonstrated value. In fact, a well-defined yet entirely manual process is preferred to an ill-defined and fully automated process.

The key parts of defining good process are as follows:

1. Identify users. Who is the process for, and what is valuable for them? The process must be oriented to their needs.
2. Define the process. Document a reliable and repeatable set of steps, develop checklists, and use a service desk or ticketing system to implement a simple workflow to capture instances of the process, their progress, and issues relating to them. No automation is required here, but it is important to ensure that the process is executed the same every time and a system for capturing metrics is in place.
3. Measure. Watch as the process is executed and identify pain points and other areas for improvement.
4. Optimize. Incrementally improve the process until it is reliable and repeatable.
5. Automate. Once enough data is available, determine the processes that have a high enough return on investment (ROI) to automate and implement automations.

It is important to understand that to justify automation there must be an expected rate of return that, spread over a reasonable period of time, is more than the cost to automate. Figure 4 below illustrates the automation decision curve. To calculate the ROI, you must first have a repeatable process in place and enough data from measuring it to understand the benefits from automating it. This is why it is important not to rush to implement automations before the ROI picture is fully understood. The natural evolution of DevSecOps practices and tools is captured in the maturity levels described below.

DevSecOps Platform Independent Model

The DevSecOps Platform Independent Model (PIM) is an comprehensive reference to fully design and execute an integrated Agile and DevSecOps strategy in which all stakeholder needs are addressed. It was developed using model-based systems engineering (MBSE) techniques to holistically define the activities necessary to consciously and predictably evolve the pipeline, while providing a formal approach and methodology to building a secure pipeline tailored to an organization’s specific requirements. The DevSecOps PIM includes a four-level maturity model that supports the mapping of current or proposed capabilities onto the set of capabilities and requirements defined in the PIM. This alignment ensures that the DevSecOps ecosystem under consideration, or being assessed, implements the breadth of best practices required to achieve a given level of maturity. The PIM defines four maturity levels where higher maturity levels build upon the practices of lower maturity levels. These maturity levels are defined as follows:

• ML1 - Performed Basic Practices: This ML represents the minimum set of engineering, security, and operational practices that is required to begin supporting a product under development, even if these practices are only performed in an ad hoc manner with minimal automation, documentation, or process maturity. This level is focused on minimal development, security, and operational hygiene.
• ML2 - Documented/Automated Intermediate Practices: Practices are completed in addition to meeting the ML1 practices. This level represents the transition from manual, ad hoc practices to the automated and consistent execution of defined processes. At this level, the pipeline includes the capability to automate the practices that are most often executed or produce the most unpredictable results. These practices include establishing processes that allow activities to be repeated.
• ML3 - Managed Pipeline Execution: In addition to performing the practices established under ML1 and ML2, practices at this level include consistently meeting the information needs of all relevant stakeholders associated with the product under development so that they can make informed decisions as work items progress through a defined process.
• ML4 - Proactive Reviewing and Optimizing DevSecOps: Practices are completed in addition to meeting the level 1-3 practices. At this level, practices include reviewing the effectiveness of the system so that corrective actions are taken when necessary and quantitively improving the system’s performance as it relates to the consistent development and operation of the product under development.

The maturity model considers the natural evolution of a good process. ML1 focuses on defining the core processes to engineering, securing, and operating software. Organizations must first understand their needs before they can automate them. This isn’t to say there is not automation at ML1, it is simply focused on the minimal set of practices one would expect to see with or without automation. ML2 is focused on creating reliable and repeatable practices in which automation can play a key role. ML3 focuses on measurement and meeting various information needs across a variety of stakeholders, followed by ML4 which is focused on optimization.

In addition to maturity levels, the DevSecOps PIM is broken down into 10 capabilities:

• Configuration management is the set of activities used to establish and maintain the integrity of the system and product under development and associated supporting artifacts throughout their useful lives. Different levels of control are appropriate for different supporting artifacts and implementation elements and for different points in time. For some supporting artifacts and implementation elements it may be sufficient to maintain version control of the artifact or element that is traced to a specific instance of the system or product under development in use at a given time, past or present, so that all information related to a given instance, or version, is known. In that case, all other variations of the artifacts and elements can be discarded as subsequent iterations are generated or updated. Other supporting artifacts and implementation elements may require formal configuration, in which case baselines are defined and established at predetermined points in the lifecycle. Baselines and subsequent changes, which will serve as the basis for future efforts, are formally reviewed and approved. The configuration management capability of a system matures with increased consistency and completeness of the integrity controls that are put in place to capture all supporting artifacts and implementation elements associated with the system and product under development while keeping pace with the DevSecOps pipeline through automation and integration with all aspects of the lifecycle. This includes (1) monitoring the relationship between artifacts and elements for a given instance, or version, of the system or product under development, (2) capturing sufficient information to identify and maintain configuration items, even if those who created them are no longer available, (3) defining the level of control each artifact and element requires based on technical and business needs, (4) systematically controlling and monitoring changes to configuration items, and (5) enforcing and logging of all required relevant stakeholder reviews and approvals, based on the organization, project, and team policies and procedures.
• Deployment is the set of processes related to the delivery or release of the product under development into the environment in which users of the product interact with it. The deployment capabilities of the system mature with increased levels of automation and advanced rollback and release functionality.
• Hosting services are made up of the underlying infrastructure and platforms that both the system and product under development operate upon. This includes the various cloud providers, on premises bare-metal and virtualization, networks, and other software as a service (SaaS) that is utilized along with the management, configuration, access control, ownership, and personnel involved.
• Integration is the process of merging changes from multiple developers made to a single code base. Integration can be made manually on a periodic basis, typically by a senior or lead engineer, or it can be made continuously by automated processes as individual changes are made to the code base. In either case, the purpose of integration is to assemble a series of changes, merge and deconflict them, build the product, and ensure that it functions as intended and that no change broke the whole product, even if those changes worked in isolation.
• Monitor and control involves continuously monitoring activities, communicating status, and taking corrective action to proactively address issues and consistently improve performance. More mature projects automate as much of this as possible. Appropriate visibility enables timely corrective action to be taken when performance deviates significantly from what was expected. A deviation is significant if it precludes the project from meeting its objectives when left unresolved. Items that should be monitored include cost, schedule, effort, commitments, risks, data, stakeholder involvement, corrective action progress, and task and work product attributes like size, complexity, weight, form, fit, or function.
• Planning and tracking is the set of practices one uses to define tasks and activities. It also includes the resources one needs to perform those tasks and activities, achieve an objective or commitment, and track progress (or lack thereof) towards achieving the given objective. It provides the mechanisms required to inform relevant stakeholders where an effort currently is within the process and whether it is on track to provide the expected outcomes. These mechanisms allow relevant stakeholders to determine what has been accomplished and what adjustments or corrective actions need to occur to account for impediments and other unforeseen issues. Ideally, impediments and issues are proactively identified and addressed. Practices include documenting activities and breaking them down into actionable work to which one can assign resources, capturing dependence, forecasting, mapping work to requirements, collecting data, tracking progress to commitments, and reporting status. The planning and tracking capability of a system matures as the automation and integration of associated practices increases.
• Quality assurance is a set of independent activities (i.e., free from technical, managerial, and financial influences, intentional or unintentional) designed to provide confidence to relevant stakeholders that the DevSecOps processes and tools are appropriate for, and produce products and services of suitable quality for, their intended purposes. It assumes that the organization’s, team’s, and project’s policies and procedures have been defined based on all relevant stakeholder needs, which will result in a value stream that consistently produces products and services that meet all relevant stakeholder expectations. The quality assurance capability of a system matures as its ability to assess adherence to and the adequacy of the defined policies and procedures improves.
• Software assurance is the level of confidence that software functions only as intended and is free from vulnerabilities either intentionally or unintentionally designed or inserted as part of the software throughout the full software lifecycle. It consists of two independent but interrelated assertions:
• The software functions only as intended. It exhibits only functionality intended by its design and does not exhibit functionality not intended.
• The software is free from vulnerabilities, whether intentionally or unintentionally present in the software, including software incorporated into the final system.

It is the responsibility of the DevSecOps system to ensure that software that meets the organization’s threshold for software assurance is allowed to be deployed and operated.

• Solutions development determines the best way of satisfying the requirements to achieve an outcome. Its goals are to evaluate baseline requirements and alternative solutions to achieve them, select the optimum solution, and create a specification for the solution. Each development value stream develops one or more solutions, which are products, services, or systems delivered to the customer, whether internal or external to the enterprise.
• Verification and validation is the set of activities that provides evidence that the system or application under development has met expected requirements and criteria. The scope includes the general realm of testing, verifying, and validating activities and matures as automation, feedback, and integration with other elements increase.

These capabilities holistically incorporate the 200+ DevSecOps requirements needed to achieve the value and mission impact illustrated in the DevSecOps continuous loop above in Figure 1. Additionally, the PIM has defined these capabilities in terms of maturity. For example, the PIM has defined Planning & Tracking Capability Maturity level 1 as Manual practices are used, with possible use of some rudimentary tools, that collect and store information used to track and report status and outputs from planning and tracking activities.

Benchmarking Your DevSecOps Capabilities

Using the DevSecOps PIM, an assessment team can evaluate an organization or program against the model’s DevSecOps requirements by considering evidence gathered, both in the form of written documentation and interviews, to determine the level for each of the 200+ distinct requirements within the PIM. Based on DevSecOps assessments the SEI has performed on numerous organizations using the PIM, we have determined the following assessment findings to be an effective way to benchmark, or take a snapshot of, an organization’s current DevSecOps maturity to establish a baseline and roadmap to continuous improvement. The four levels of the scale of findings are:

• Consistently Demonstrated
• Occasionally Demonstrated
• Insufficient Evidence Demonstrated
• Not Applicable

Using this scale, one can produce a summary benchmark such as that shown in Figure 5.

When focusing on value, a key element of the scale is Not Applicable. A requirement or activity may be called out in the PIM as a best practice in DevSecOps, but that doesn’t necessarily mean it is relevant to the organization being assessed. If a given requirement within the PIM doesn’t drive value through mission impact, then it should be discarded as Not Applicable.

The DevSecOps PIM Maturity Model can be used to

• provide awareness of what practices are already in place based on a holistic set of Agile and DevSecOps requirements and identify practices that are not applicable
• identify pain points, barriers to collaboration, and technological barriers with respect to DevSecOps and Agile principles
• propose areas of improvement and strategy regarding implementation of software development tools and methodologies that seem applicable to the program’s mission set

The goal of using the DevSecOps PIM is not to establish an ideal Agile or DevSecOps state. The goal is to identify actions that an organization, and those in their orbit, can take to make assessments and, on this basis, evolve into a more effective and efficient organization that delivers increased value for future engagements.