Most of the software in use today is assembled from existing code connected with third-party services and products. Reuse is extensive, which makes it faster and cheaper for developers to field systems without starting from scratch. The downside is that this reused code contains defects unknown to the new user, which, in turn, propagate vulnerabilities into new systems. We see the primary focus from system design on new code, and organizations are turning to DevSecOps to produce it faster and at lower cost, but the reality is that much of the code is actually coming from the software supply chain through code libraries, open source, and third-party components. These sources are troubling news in an operational climate already rife with cybersecurity risk. Organizations must develop a cybersecurity engineering strategy for systems that addresses the integration of DevSecOps with the software supply chain.

In this blog post, I build on ideas I presented during a recent webcast about the challenges of cybersecurity when integrating software from the supply chain. I will first explore the challenges of building cybersecurity into systems that rely on the software supply chain and must function within the current software-enabled threat landscape. Then I will follow by introducing considerations for implementing a cybersecurity engineering strategy to meet these challenges that ties the DevSecOps pipeline with the realities of the software supply chain.

Emerging Cybersecurity Needs in the Software Supply Chain

The supply chain of reused software code introduces several issues that must be considered by acquirers, program management, and engineers. Start with the basic understanding that all suppliers have their own processes and practices for managing development and cybersecurity. Each piece of reused software blends new and existing code aimed at meeting a set of requirements. These requirements may differ substantially from those for the planned reuse. Differences in the cybersecurity aspects of the original requirements will impact the risk from the code in reuse.

All software carries some level of defects, which varies depending on the code quality. Research has shown that an estimated 5 percent of these defects can become vulnerabilities, but each piece of code has a different owner that may or may not be fixing the potential vulnerabilities in a timely fashion. Moreover, every integrator must incorporate the fixes into their system before they can reduce the potential impact.

Once code is selected for reuse, the systems integrator has varying degrees of control over this code depending on many factors, including acquisition strategy. Is source code available and does the acquirer have resources sufficient to take ownership should a problem arise? Will the original builder of the code retain control and provide updates as they see fit, and is the integrator prepared to apply these updates? Has consideration been made for potential risk resulting from missing or delayed corrections? This code-risk analysis must be replicated with the introduction of each new software-intensive product.

Code quality is a large factor in the level of defects to manage. According to Capers Jones’s research, “best in class” code has fewer than 600 defects per million lines of code while “good code” has fewer than 1,000 defects per million lines of code. Finally, “average” code has 6,000 defects per million lines of code. Our own research found that some portion of security vulnerabilities (maybe more than 50 percent) are also quality defects. Improving software quality by reducing the number of coding defects or errors also reduces the number of vulnerabilities and therefore improves software security.

Few organizations have adopted practices for effectively managing reuse within the software-development lifecycle. Most see reused code as free. However, organizations developing new software by building on top of existing code may also be shepherding functionalities into the new system that may no longer be relevant. Different products map to desired functionalities, but each component is a decomposition of code that is collected from subcomponents, commercial products, open source, code libraries, and so on. Each of these code components collects, stores, and sends data in different file structures and formats, and far too often no one person on the integration team can understand or manage how all these pieces fit together.

Another complicating factor is that when software patches are introduced to address vulnerabilities, those in charge of integration must select what updates they apply and then contend with potential incompatibilities that can impact the operational execution of the updated system. If they lack transparency into what is included in their integrated product, also referred to as a software bill of materials (SBOM), the risk of a critical patch being missed is high.

Many organizations struggle to address these ever-increasing cybersecurity challenges. Too often they allocate only operational resources to react to problems after those potential vulnerabilities enter into operational execution. Adoption of incremental development and a DevOps approach integrating development and operations provides an opportunity to proactively search for and address these potential vulnerabilities in advance. However, the workload of the pipeline must be structured to prioritize analysis of existing code along with new functionality.

The pace of implementation and the expanded use of automation encouraged in this approach requires closer integration of cybersecurity into both parts of the lifecycle, hence DevSecOps. Resources must be applied throughout the lifecycle to establish and deliver effective cybersecurity, which the supply chain further complicates.

An effective cybersecurity engineering strategy can provide the plan for closely coupling all these components. When the supply chain is a major provider of product capability, the plan must consider the ways problems can be introduced from the supply chain and how resulting potential vulnerabilities will be addressed. Since the supply chain components were developed to a different set of requirements, product testing alone will be insufficient if the focus is on verification of requirements. Support from each supplier can add value as input if available, and continuous code scanning of source and binary objects must be fully integrated into pipeline activities.

Elements of a cybersecurity engineering strategy should include the following:

• Establish security requirements to ensure confidentiality, integrity, availability (CIA) for developed code, as well as reused code.
• Monitor the pipeline and product for CIA including supply chain considerations for both.
• Implement appropriate lifecycle processes and practices in the pipeline structure and the product integration to reduce operational vulnerabilities in both the developed and reused code.
• Establish coordination and communication capabilities among the many participants, including the supply chain, to ensure timely and effective response.

Using this view of the challenges that the supply chain presents for cybersecurity, I will explore in the remainder of this post how to deploy a cybersecurity engineering strategy to address these software-linked supply-chain issues with the DevSecOps pipeline.

Engineering the DevSecOps Pipeline Integration with the Supply Chain

The DevSecOps pipeline is a social-technical system composed of both software tools and processes. As the figure below illustrates, as the capability matures, the DevSecOps pipeline can seamlessly integrate three traditional factions that sometimes have opposing interests:

• development, which values features
• security, which values defensibility
• operations, which values stability

A DevSecOps pipeline emerges when continuous integration of these three factions is used to meet organizational, project, and team objectives and commitments.

Each of these areas is assigned to different parts of the organization, so coordination is imperative. Automation will not replace coordination. In our work with government organizations, we often encounter groups that have implemented a pipeline and automated sections of it, but many of the recipients that need information from the automated processes do not receive it because they were not part of initial plans. The pipeline can collect lots of data about cybersecurity, but if appropriate monitoring and managing of that information is not implemented to address cybersecurity effectively, the results will be not as expected.

Organizations must consider the following supply chain issues when developing and implementing a DevSecOps pipeline:

• Too often organizations focus only on cybersecurity considerations for the developed code, which is insufficient given the level of reuse that impacts current products.
• Automating existing practices and processes requires all the various parts of the organization (i.e., operators, developers, managers) to work together with the pipeline suppliers, which provide infrastructure components, tooling, and sometimes parts of the product.
• The automated pipeline itself represents a system that also includes reused code and components and thus should be engineered to address cybersecurity effectively with its supply chain.

Pipelines do not spring up out of the box fully implemented. The maturity process that increases functionality, capability, and coordination is the result of continuous monitoring and improvement. We have identified four levels of maturity that evolve the pipeline from basic execution of steps into initial automation, managed execution, and finally proactive execution. The degree to which cybersecurity is embedded will increase with each level, but since the pipeline is an integrated system that is constantly changing, how well it works must be monitored and managed continuously. Supply chain considerations will require pushing cybersecurity maturity considerations into supplier behavior.

Four Different Levels of Maturity in the Cybersecurity Pipeline

Through our work, we have identified four different levels of maturity in the cybersecurity pipeline that reflect the increased functionality that comes over time from implementation and continuous monitoring and improvement. Suppliers are not described specifically since their interactions will vary based on how the cybersecurity strategy defines their relationship with the pipeline. But they are active participants in the processes, and their actions must support the increased maturity.

Planning for how the different parts of the acquisition and development lifecycle will integrate is critical to reaping the benefits of the DevSecOps pipeline and avoiding operational aggravations and additional risk. The complexity of the DevSecOps environment must also be taken into account. Business requirements drive the unique needs of each organization. Moreover, the product and infrastructure, which are often considered as different pipelines, need to work in concert. Interactions with each supplier providing components, tools, and services for both the product and the pipeline must be part of this plan.

As noted earlier, organizations often focus almost exclusively on new code that they are developing, but they do not consider the inherited risk that reuse introduces when defining the integration of shared services, open-source software, and third-party products into the pipeline. In some cases, technology approaches such as containerization are selected to solve the risks coming into the pipeline from third-party sources. This approach represents expanded use of supplier-provided capabilities and is not a solution independent of the operation of the pipeline. As more automation is incorporated into the pipeline that executes supplier-supported capabilities, sufficient measures and reporting must be in place to continuously justify the level of trust. Continued assurance that the pipeline and its products maintain CIA and that vulnerabilities are addressed must be demonstrated, monitored, and managed and not assumed.

Some organizations architect the product externally and then feed detailed requirements for software development into the pipeline. Other organizations deliver only software out of the pipeline that feeds into integration with specialized hardware and specialized testing for compliance before operational use. The pipeline can be different parts of the lifecycle, depending on what the organization needs to deliver.

Each one of these approaches imposes different cybersecurity requirements on the DevSecOps pipeline. Whatever the role of the DevSecOps pipeline, effective cybersecurity requires coordination among acquisition, engineering, development, infrastructure, and security. Effective management of the pipeline and the product requires a focus on how all of these pieces fit together, including the supply chain.

To support a more seamless integration of the supply chain with engineering, program management, and the DevSecOps pipeline, for the past year, I have been working with a team of researchers in the SEI’s CERT Division to develop an Acquisition Security Framework (ASF). The ASF captures a baseline set of processes and coordination practices that should integrate with each pipeline for effective cybersecurity. In a future post, I will present this framework, which will allow organizations to compare current practices with what is needed to identify potential gaps that could represent supply chain risk.