Historically, software security has been addressed at the project level, emphasizing code scanning, penetration testing, and reactive approaches for incident response. Recently, however, the discussion has shifted to the program level to align security with business objectives. The ideal outcome of such a shift is one in which software development teams act in alignment with business goals, organizational risk, and solution architectures, and these teams understand that security practices are integral to business success. DevSecOps, which builds on DevOps principles and places additional focus on security activities throughout all phases of the software development lifecycle (SDLC), can help organizations realize this ideal state. However, the shift from project- to program-level thinking raises numerous challenges. In our experience, we’ve observed five common challenges to implementing DevSecOps. This SEI Blog post articulates these challenges and provides actions organizations can take to overcome them.

Key Benefits of DevSecOps

The addition of security to the practice means addressing security throughout the lifecycle, from the concept of a feature to the deployment of a product. The need for common principles and practices that span the organization can itself present a challenge. In addition to principles and practices that span the organization and govern the lifecycle from concept to deployment, DevSecOps requires the following practices:

• iterative and incremental development—This practice involves a cyclical approach to breaking software development into smaller, more manageable steps. Continuous integration/continuous deployment (CI/CD) practices provide the automation necessary to ensure quality, security, and functionality.
• continuous feedback—Feedback should be collected throughout every step of the lifecycle to allow for expert validation and general observability. Every tool used throughout a DevSecOps pipeline creates some output that can be used for feedback. Test cases produce output that provides quality feedback, stakeholders and customers offer a source of human feedback. Figure 1 illustrates the kinds of data, and their sources, that can inform the feedback and measurement cycle.
• metrics and measurement—Metrics are used to evaluate feedback and determine how well the organization is performing on measures, such as productivity and quality.
• automation in every phase of the Software Development Lifecycle (SDLC)—Automation helps organizations make the most of their feedback mechanisms. CI/CD is the primary automation mechanism of the SDLC.
• complete engagement with all stakeholders—All stakeholders must be engaged, not just the Dev, Sec, and Ops components. This engagement should produce consensus on what matters the most to the organization.
• transparency and traceability across the lifecycle—Transparency helps build the trust needed among the Dev, Sec, and Ops teams to make the process work, and traceability enables the digital trail of the products used to build, deploy, and maintain software.

These practices produce several benefits when applied in DevSecOps. Perhaps the two most important are the following:

• reduced security error and associated costs—By addressing security issues throughout the development lifecycle rather than after release, organizations can catch and address issues earlier, when the time and cost to resolve them is much lower. These costs include lost dollars, additional effort and rework, and customer goodwill.
• reduced time to deploy—Catching flaws and vulnerabilities through constant testing during the lifecycle reduces time to deploy, reduces time spent after deployment on error response, and improves your readiness to deploy.

In addition to fewer errors and vulnerabilities, reduced costs, and reduced time to market, DevSecOps also provides the following benefits:

• repeatable and/or automated steps
• continuous availability of pipeline
  and application
• increased time for learning new concepts
• responsiveness to business needs
• increased stability and quality

DevSecOps Challenges

While DevSecOps can benefit your organization in many ways, we’ve observed several challenges to its adoption, the most common of which are the following:

1. lack of security assurance at the business and project levels
2. organizational barriers related to collaboration, tooling, and culture
3. impact to quality because security is not a priority while systems are getting more complex
4. lack of security skills for developers, business stakeholders, and auditors
5. insufficient security guidance due to lack of resources, standards, and data

The rest of this section examines each of these challenges and provides approaches for overcoming them.

CHALLENGE #1: Lack of Security Assurance

How do we know that the security practices we’ve adopted for our development lifecycle and built into our software are adequate and appropriate for the business purpose we’re trying to address? Addressing this challenge can be hard, especially when your industry, business, and/or project lacks security assurance.

Your Industry Lacks Security Assurance Models

The security requirements for your industry or domain are different from those of other industries or domains. For instance, the health care industry has different security requirements from the financial sector. If your industry lacks assurance models, you might begin by assessing your own organization’s security posture and requirements, then engage with similar organizations in your industry or domain to share information. In addition, we recommend the following:

• Don’t wait for an industry standard to emerge.
• Join or create informal working groups with industry peers.
• Attend conferences and network with like organizations.
• Share your experiences and lessons learned.
• Work with others to extend the body of knowledge and establish best practices.

Your Business Lacks Security Assurance

There is often a disconnect between business needs, mission, and vision when it comes to security. Developers need to understand the business context of security. They must think about the organization’s security policies, its business drivers, and how these apply to the software being developed. In so doing, they should address security as early as possible in the lifecycle, preferably during the requirements stage. As you do, keep the following recommendations in mind:

• Focus on fundamentals: What are the threats? What are the business drivers? Balance the two.
• Align with development with business needs (time to market, cost savings, resilience).
• Conduct external audits.
• Understand the business context.
• Identify, link, and rank business and technical risks.
• Identify security requirements in early.
• Define the risk mitigation strategy.
• Educate top management and get them onboard.
• Engage more senior technical people first to work with security teams.
• Make security part of senior technical reviews; organically spread the word.

Your Project Lacks Assurance of Security

Once you’ve identified security assurance needs within your industry or domain (and perhaps your specific business within that domain), you need to map that data to your project. For instance, perhaps your organization is already following guidance, such as General Data Protection Regulation (GDPR) or the Health Insurance Portability and Accountability Act (HIPAA). You need to account for any security activities stipulated in that guidance in your project planning, and you need to do so early in the lifecycle when there’s still time to address it. As you do, keep in mind the following recommendations:

• Map reporting data from tools to form a continuous view of value.
• Run security tools on all code to measure code quality and standards.
• Review code changes for security and document approval prior to release.
• Use dedicated testing resources in the case of significant changes.
• Track all changes and approvals for incident purposes.
• Conduct code reviews.
• Expose security team to your metrics and data.

CHALLENGE #2: Organizational Barriers

If you’re not sharing your DevSecOps journey across the organization, from concept to product, you should expect problems since you won’t have a clear understanding of the business needs your software needs to address. You might not even have a clear vision of the customer’s needs and the environment in which the customer operates. Communication is key to breaking down organizational barriers, but often different units inside an organization use different communications tools, structures, and goals.

To begin to break down these barriers, briefly document your journey from idea to product. Look at points of interaction among the various components of your organization. Educate executives who likely don’t know the details of the DevSecOps process. Build connections and a culture that reinforce the sharing of the same goals and vision. Often, poor stakeholder collaboration, difficulty integrating pipeline security, and an unwillingness to make security a priority stand in the way of successful DevSecOps implementation.

Poor Stakeholder Collaboration

The product you’re creating touches many other stakeholders in your organization, including marketing, IT, and legal teams, but communication among them might be lacking. For example, you may have different tools, may not share the same infrastructures, and may not even share the same vision and goals. To address these issues, you need to come together and document your journey from concept to product. A simple cheat sheet will suffice, one that reminds all stakeholders of the vision, the mission, and the roles they will play in the lifecycle. Our recommendations for improving stakeholder collaboration include the following:

• Document your current state and identify silos (e.g., development, infrastructure, and security).
• Start building collaboration between the Security, Dev, and Ops teams.
• Be prepared: people generally don’t want to change their culture and/or workflow.
• Make sure everyone gets on the same page regarding the importance of security (from executives to DevSecOps teams).
• Instill a continuous security mindset.
• Focus on partnership, not unhealthy conflict. Destroy the blame culture.
• Get stakeholders to agree on a shared a vision for the project.
• Balance workload among teams involved.
• Put security people into development teams.

Integrating Pipeline Security

The pipeline is not only the infrastructure supporting DevSecOps. Instead, it is the heartbeat of your entire DevSecOps ecosystem, including source control, communications, issue tracking systems, documentation systems, CI/CD, and code review. This infrastructure should be connected, i.e., all the tools should communicate to each other, as shown in Figure 2.

For instance, your source control should be able to communicate with your build server, your communication systems, and your issue tracking systems. When your infrastructure is connected this way, you can apply threat modeling, static analysis, dynamic analysis, project management, or interactive application security analysis. Think about tool integrations to overcome pipeline security problems and then design your infrastructure to address security.

The pipeline is where transparency happens and where all the stakeholders enforce their expertise via automation. One way to achieve this transparency is through metrics dashboards fed by pipeline data that are easy to read. The tool should be tailored to the product. The bigger you are, the harder this is to do, but the result is worth it. Recommendations for integrating pipeline security include the following:

• Integrate your process with threat modeling (TM), static application security testing (SAST), dynamic application security testing (DAST), and interactive application security testing (IAST).
• Install security requirements traceability.
• Apply metrics: mean time to repair (MTTR), mean time to detect (MTTD), vulnerability escape rate, repeated incident root cause, time to deploy the app from development to production.
• Look at different approaches: abuse cases, architectural risk analysis, application penetration testing.
• Design for security.
  • fail securely and fail safe defaults
  • least privilege
  • defense in depth
• Automate where possible.
  • infrastructure as code (IaC), virtualization, containers, and load balancing
  • configuration management
  • continuous application and performance monitoring

Making Security a Priority

You need to plan for security if you want to make it a priority. Treat security as a feature that your software must have. In some cases, the choice is out of your hands: a software bill of materials (SBOM), for instance, might mandate building security into the product. But how do you make security a priority? We recommend the following:

• Use evangelists to drive culture change.
• Explain why security is an important, shared responsibility, and its impact.
• Embed security into operations escalation.
• Invite the security team to postmortems.
• Create a plan in small parts; start with a pilot and be mindful of cross-team resource constraints.
• Keep it simple; don’t overwhelm the system. If there are too many things to do, the plan is likely to fail.
• Incrementally chase real risk and threats first.
• Test whether your organization is ready for the culture change; no single technology/tool will get you DevSecOps.

CHALLENGE #3: Lack of Quality

Security is integral to quality. In our observation, lack of quality is often associated with the security team getting involved too late, a lack of confidence in the release, and system complexity.

Security Team Involved Too Late

Too often, developers make security a secondary priority, especially when they are under pressure to keep production moving forward. As we stated earlier, security is a key aspect of quality. When the security team engages towards the end of the SDLC process, it’s often too late to avoid the disruption and expensive rework that flows from the security flaws it identifies. Depending on the project cadence, “too late” may mean two months, two weeks, or even two days.

Consider a team using Agile development with two-week sprints. Within that sprint, given a scrum every day, the developer would need to know of any problems as early as possible. However, the Sec team only analyzes the code a month later (or maybe two months later or just before deployment to the production environment). Any problems discovered by the Sec team at this point will require tremendous work, and developers will push back. Moreover, the later problems are discovered in the SDLC, the more expensive they are to fix. To avoid these issues, we recommend the following:

• Start getting security and compliance requirements in early.
• Tie compliance objectives into providing assurance back to the business.
• Test compliance against security policies to identify gaps.
• Define a risk mitigation strategy early.

Lack of Confidence in the Release

Correcting problems and patching security vulnerabilities late in the development lifecycle when the pressure is on to get the product out the door opens room for doubt about the quality of your release. This lack of confidence hinders planning and effective use of resources as you need to reserve resources to address flaws and vulnerabilities discovered after release. These flaws and vulnerabilities represent an opportunity cost, a dollar cost, and a reputational cost. But there are ways to improve confidence in your release, including the following:

• Instill risk-based security testing.
• Move the conversation from CABs and phase gates to compliance driven releases.
• Automate reporting for compliance violations and stop the pipeline when the threshold is exceeded, or policy not met.
• Move toward frequent, automated audits.
• Audit yourself to demonstrate compliance with policies or regulations.
• Establish security requirements traceability (a feature DevOps provides) and trace everything: code, artifacts, pipeline, test cases, etc.

System Complexity

Consider a complex system with multiple application programming interfaces (APIs) and microservices. How do you gauge its quality? How do you know that each of the services is following the right security controls? How do you know that each API is following centralized communications? How do you know that they are following the security policies that you enforce in organizations? You need to incorporate these policies in your code, in your architectures, in your microservices. To do so, you need to collect the right data and metrics that enable you to examine all the components throughout your complex system. The more complex your system, the more you need a testing environment that mirrors your production environment. In short, we suggest the following:

• Identify proxy metrics for complexity, such as the number of issues in production and the time to deploy an application.
• Drive security policies into production by integrating security tasks in early stages of the DevSecOps pipeline.

CHALLENGE #4: Lack of Security Skills

Developers, architects, scrum masters, and other key players in an organization should have the right vocabularies and skills. By vocabularies, we mean some common knowledge or skillset, or a common understanding, such as a knowledge of how to write secure code. In our experience, this lack of a common vocabulary often manifests in three ways: The business lacks security skills, developers lack security skills, and/or auditors lack security skills.

The Business Lacks Security Skills

Business stakeholders need to use the vocabulary of security. Of course, not everyone can be an expert at everything, but the business stakeholders should be able to understand and articulate security requirements and connect those security requirements to organizational risks. An acquisition team, for instance, should be able to know it’s acquiring the right security practices when it’s acquiring a product. To improve in this area, we recommend the following:

• Shift the conversation to risk and quality.
• Service and protect the business interests to lower risk (identify risk and/or security value).
• Identify architectural risk and uncertainty and map these risks to compliance, resiliency, and feature delivery.

Developers Lack Security Skills

We like to think that developers know everything needed to perform their tasks successfully. Developers certainly know how to write code, but they are often not trained for secure coding in specific languages at the university level or in skills development programs, where the focus remains on feature development. It takes time to learn these skills, and to practice using them, but it’s worthwhile for the organization to develop these security skills among its staff, to develop. This upskilling can take the form of security training or other programs and resources that incentivize and motivate development staff to acquire and develop these skills.

You can start small with a narrow focus. For instance, ask yourself, “What are the top 10 vulnerabilities we have addressed in our organizations? What are the top 10 CVEs? What are the top 10 CWEs?” Focus on training that addresses those issues and widen your scope over time. Or start with the programming language(s) used by your organization and focus security training on those languages. Other recommendations for building security know-how among your development staff include the following:

• Keep the endgame in mind and build a collaborative security culture.
• Implement compliance automation to drive business thinking into the SDLC.
• Don’t make security training a checkbox. Security training once a year has limited effectiveness.
• Aim for attitude and behavior: Simply providing better technical training alone won’t change attitudes.
• Motivate and unblock the path to your goal: Remove task ambiguity, set clear role targets, and don’t overload.
• Aim for long-term retention and apply learning in context repeatedly.
• Rotate security experts on the team, where possible.

Auditors Lack Security Skills

Auditors review code and products, rendering a thumbs-up or a thumbs-down based on established criteria, but they generally don’t have the skills and knowledge to provide an accurate judgment about security. To compensate, foster strong relationships among auditors and developers. Educate auditors on your architecture and systems. As we noted earlier in the section on business stakeholders, make sure your auditors understand and use the same vocabularies. Other recommendations include the following:

• Build working relationships and collaboration across silos.
• Make security a part of informal discussions.
• Provide cross-functional training for both technical and compliance domains.
• Integrate low-disruption workflows.
• Get familiar with some common standards and frameworks (OWASP Top 10, NIST 800-53, and ISO 27001).

CHALLENGE #5: Insufficient Security Guidance

Organizations need to take stock of their compliance practices and security policies and implement them in their products or capabilities. For instance, you may have adopted zero trust policies, but how will you implement them? Consider where your organization is in its DevSecOps journey and map out your future: What are you doing for year one? Year two? Beyond? Remember, if you want to create a new standard for your organization, you might think you’re unique, but you’re not. Instead of starting from scratch, use an existing standard or try to tailor one to your needs. For instance, you might start with the OWASP Top 10. How can you implement those frameworks in guidance under CI practices? Incorporate the policies into the workflow from beginning to end.

In our experience, problems in this area stem from three deficiencies: lack of security resources, lack of security standards, and/or lack of proactive monitoring.

Lack of Security Resources

You probably don’t have enough security people on the team, yet there are policies and guidance to develop. Moreover, there’s so much else that must happen. If you’re lucky, you might have some unicorns on your team, some overachievers, but you need to get beyond that. Here are some recommendations for organizations that want to embark on a DevSecOps journey but lack security resources:

• Start small by introducing a policy and assess your gaps. Grow from there.
• Map policies to domain-specific procedures (e.g., development and testing) and implement in product (e.g., zero trust).
• Aim for long-term sustainability: When you evaluate an upgraded capability a couple of years after deployment, is the change still there?
• Spread security responsibility across multiple people.

Lack of Security Standards

Here’s the good news: As we’ve noted, several security standards have already been developed. You don’t have to start from scratch. You can start with policies derived from existing standards, tailor them to your needs, and incorporate them into your practices and products. When doing so, keep these recommendations in mind:

• Don’t go big bang. Start with something manageable, such as static analysis, and grow from there.
• Start with a well-known framework like OWASP Top 10 and create a few policies derived from that.
• Aim at low hanging fruit (CI or testing, for example) and measure security against your initial policies.
• Grow by looking upstream and downstream for the next-easiest implementation.
• Bake policies into the workflow to avoid regression.

Lack of Proactive Monitoring

Proactive monitoring [DS1] identifies and addresses security risks before an attack happens. The key to developing more proactive monitoring is to look at cases in which you’ve been forced to react, then plan ways to get in front of the problem. For instance, you may have discovered certain vulnerabilities, or classes of vulnerabilities, after past releases. Think about how you can address those kinds of vulnerabilities in your pipeline and develop a practice around that and then incorporate it as early as you can in your SDLC.

There are great open-source and commercial monitoring tools available. While each of these tools has an individual dashboard, ideally pe the results from all of them should be integrated into a common dashboard. Doing so will provide an overarching view into your DevSecOps journey for both the organization and your individual products. We also recommend the following:

• Start with Ops log monitoring before attempting expensive tools.
• Create feedback loop from Ops back to development.
• Update security documentation, including trust boundaries, new threats, and component verification.

Conclusion: Sustaining your DevSecOps environment

Implementing DevSecOps can be daunting. Challenges abound. This blog posting examined the five most common challenges and approaches for overcoming them, but perhaps the overarching challenge is the scope of the task. In particular, you want to realize the benefits of DevSecOps, but you worry your organization lacks the resources and know-how to achieve a fully realized DevSecOps environment. The prospect can be overwhelming, which is why throughout this post we have characterized the process as a journey and recommended starting with small steps you can manage and build on. As you do so, keep this cyclical process in mind:

1. Assess your gaps.
2. Identify quick wins.
3. Empower champions and highlight accomplishments.
4. Measure results, reassess gaps, and build on quick wins.
5. Evaluate and repeat.