The benefits of operating a development environment with continuous-integration and continuous-delivery (CI/CD) pipeline capabilities and DevSecOps practices are well documented. Leveraging DevSecOps practices and CI/CD pipelines enables organizations to respond to security and reliability events quickly and efficiently and to produce resilient and secure software on a predictable schedule and budget. Although the decision by management to adopt this methodology may be easy, the initial implementation and ongoing improvement of the methodology can be challenging and could result in incomplete adoption or ineffective implementation.

In this and a series of future blog posts, we provide a new framework to guide organizations in the planning and implementation of a roadmap to functional CI/CD pipeline capabilities.

This framework builds on well-established applications of DevSecOps principles and provides additional guidance for applying DevSecOps principles to infrastructure operations in an on-premises computing environment by providing an ordered approach toward implementing critical practices in the stages of adoption, implementation, improvement, and maintenance of that environment. The framework also focuses on the leverage of automation throughout the process.

The practices we outline are based on real experiences supporting on-premises development environments tailored to the missions of the sponsoring organizations. The framework we define here can also be adapted and applied to the process of leveraging commercially available cloud services to deploy a development environment.

While other sources have described various stages of the DevSecOps progression, such as the 5 Stages of DevOps Evolution outlined in the 2019 State of DevOps Report, they usually focus on the practices of development teams as they release their software to their customers, on operations teams responsible for the support of the software products developed in-house, and on the management perspectives that oversee such business activities. In so doing, these sources gloss over, omit, or ignore the difficulty associated with establishing the underlying infrastructure and tooling required by these teams to achieve the full Agile and DevSecOps visions. Our framework takes a different perspective, largely ignoring the developer and management perspectives that are already well defined in literature. Instead, we focus on the development and operations teams and their leverage of DevSecOps principles and tactics while implementing the CI/CD pipeline capabilities.

To contextualize the framework we have devised, it is important to underscore the value of CI/CD capabilities. The DevSecOps CI/CD pipeline is a socio-technical system composed of both software tools and processes. Ideally, it seamlessly integrates three traditional factions that sometimes have opposing interests: development values features, security values defensibility, and operations values stability. A DevSecOps CI/CD pipeline emerges when continuous integration of these three factions is used in conjunction with continuous delivery, as shown in Figure 1 below.

The usage of a CI/CD pipeline is the cornerstone of a mature and robust DevSecOps environment. Likewise, a fully automated CI/CD pipeline is a best practice of Agile methods. From a practical standpoint, a CI/CD pipeline provides for the automatic vetting, building, testing, packaging, and delivery of a change to the desired destination. A problem in any of the stages of the pipeline is reported and fed back to the initiator of the change that caused the problem. The tools and processes used to accomplish the delivery may also be used to monitor and report on the status of any changes to a system.

A simple CI/CD pipeline involves these steps:

1. A patch or new feature for a product is approved for implementation and assigned a priority for implementation.
2. A human implements the patch or new feature and submits changes to a product in a version-control system.
3. Submitted changes initiate automated build processes for the product.
4. If build processes are successful, automated tests are executed on the product.
5. If all tests are successful, a release package is produced.
6. If packaging processes are successful, the release package is delivered manually to the production systems.
7. Package deliveries initiate automated installation or upgrade processes for the product on the production systems.
8. Over time, operations personnel implement and automate monitoring capabilities for the product.
9. A human makes an observation about the product's functionality and requests a patch or a new feature in the product. This request is fed back to step 1.

Pipeline complexity may vary widely among organizations, and even between different teams within an organization, depending on factors such as the security requirements, available hardware and software resources, and the needs of the various stakeholders.

Considering the perspectives of those responsible for implementing the CI/CD capabilities, there is significant overlap with the traditional use of CI/CD pipelines. The following benefits refer specifically to those responsible for implementing the development environment. A CI/CD pipeline provides developers with consistent and immediate reporting of the validity of code changes in a software product. Likewise, a CI/CD pipeline provides operations teams with information about the efficacy of changes to system and application configurations. Armed with this information, development and operations personnel are better equipped to communicate project statuses, which can lead to improved collaboration among team members and across teams. This ultimately leads to enhanced software quality, better infrastructure resiliency, and increased throughput of value-add features and capabilities being delivered.

The CI/CD pipeline provides security teams with insight into changes to code, infrastructure, services, and applications. The capabilities that enable CI/CD pipelines lay the groundwork for the automation of metrics gathering and report generation. Reports can be disseminated quickly and escalated when necessary so that actionable events are amplified and given the attention they require. Incident-response procedures are streamlined by these capabilities and can even leverage the same collaboration tools used by development and operations teams.

Finally, the capabilities that enable the development and operations teams to monitor and track changes to their systems and applications effectively can also be leveraged to provide management with the data required to make informed decisions about the organization's investment in the on-premises or cloud-based computing environment, and in the efficacy of the development, operations, and security of the environment.

No standard implementation of a DevSecOps environment or a CI/CD pipeline exists that works for everyone. Each organization must therefore analyze its existing culture, its desired culture, its budget constraints, and its defined business cases, and use that information to select the implementation components that best advance its mission. Remember that a key goal of CI/CD pipelines is automation and a key goal of DevSecOps is to alert someone to a problem as early in the automated-delivery process as possible so that that person can intervene and resolve the issues with the automated processes. Some business cases to consider include security constraints, such as limited network access, strict physical-access requirements, and heightened system-security requirements. These considerations are critical during the planning phases of the implementation as the organization defines its development environment's desired end state.

If your organization has decided that running your development environment on premises is a requirement to your mission, then your organization can build out an environment specifically tailored to meet those needs. If your organization has decided to leverage a commercial cloud provider, you will be constrained by the features provided by the vendor, but will not have to bear the responsibility of maintaining that infrastructure. In either case, it is extremely important to have a well-defined end state in mind before implementation begins in order to avoid costly mistakes.

When defining and establishing a vision for the environment, designers should consider the ideal state of the development environment from the perspective of all stakeholders, including software architects, developers, system administrators, test and quality-assurance engineers, security officials, management, and end users.

From the perspective of the software-development team, the ideal development environment may include characteristics such as

• access to the internet
• access to code-repository, build, test-automation, and planning tools
• access to a variety of open-source and commercial integrated development environments (IDEs)
• access to commercial and open-source software repositories
• access to a self-service interface where computing resources (virtual machines or containers) can be allocated with a selection of operating-system types and almost limitless memory, processing power, and disk space
• full administrative control over computing resources used for development

From the perspective of the operations team, the ideal environment may include characteristics such as

• a standardized set of physical hardware systems
• a standardized set of operating systems
• a standardized way of installing software
• the ability to restrict access to various network locations, applications, and services
• a robust monitoring and alerting infrastructure
• a test environment that mirrors the production environment

From the perspective of the security team, the ideal environment may include characteristics such as

• robust physical-access controls
• well-defined change-management procedures
• the ability to attribute actions to specific individuals
• the ability to track security controls for each delivery
• security-compliance metrics
• security-alerting capabilities
• automatic vulnerability remediation
• well-defined incident-response procedures

The management perspective may require such considerations as overall system-health metrics, including

• lead time that indicates responsiveness and will be affected by quality, productivity, and resource utilization
• deployment frequency that indicates how often a new release is deployed into production
• availability and time to recovery that indicates reliability
• production failure, operational errors, and rework that indicate quality issues
• exploits and attacks that are a lagging indicator of security issues
• cost of operation, including capital expenditures and effort expenditures

These needs must be balanced with the available hardware and software technologies. Functionality is a foundational element of the environment, which means it is important to ensure that the overall system architecture of your environment is well suited to your end goal. Care must be taken to select physical hardware, operating systems, platforms, and software that are compatible with each other. Cost and feature sets should also be considered. There will be tradeoffs to consider and compromises to be made. These requirements hold true for both on-premises and cloud-hosted computing environments.

After you determine the desired end state for your development environment, you can start planning the roadmap that defines how you will achieve that end state. The to-do list will be substantial, and may be intimidating! In the true spirit of Agile, you must be ready to adapt as requirements change during the environment-deployment process. You will consider all tasks related to hardware setup, networking, system and service deployment and configuration, security compliance, automation, and ongoing maintenance and operation. During your roadmap planning, you will decide which features will be available early in the environment rollout and which will be released later.

The framework we have devised supports the ultimate goal of adopting DevSecOps practices that are mature enough to support fully automated CI/CD pipelines. It accomplishes this by outlining stages that progress from basic DevSecOps practices to more advanced ones. You will notice some similarities to the progression in the 5 Stages of DevOps Evolution as defined in the 2019 State of DevOps Report. However, as we get into the details, this framework is tailored to the deployment and operation of a development environment and contains practices specifically aimed at the development and operations teams charged with managing this environment.

The stages of our framework are

0. building the foundation
1. normalization
2. standardization
3. continuous integration and continuous testing
4. infrastructure as code and continuous delivery
5. automated security and compliance as code
6. automated compliance reporting and vulnerability remediation

The first important point to note is that each stage builds on the preceding stage as your organization improves its DevSecOps capabilities. Therefore, you will benefit the most by creating a roadmap that reflects the progression from stage to stage. More mature DevSecOps environments will ultimately implement the practices in the later stages.

The second important point to note is that the sequence of the stages advances the practices from manual to repeatable to automated. You will begin by performing tasks manually, then making those tasks repeatable, and finally automating them. Ultimately, you should strive to automate as many tasks as possible. Your roadmap should reflect this dynamic as well.

Finally, as you work through the stages of the framework to establish the roadmap for the deployment of your computing environment, you must continually be mindful of the cultural requirements of DevSecOps practices. Significant shifts in the culture include better communication among the teams, a commitment to collaborate on solving issues that impact all teams, and a shared feeling of responsibility for the delivery of the end products. All stakeholders--management, developers, operations, security--must be on board with the implementation of these cultural changes to do their part in implementing them. The roadmap must reflect this. Fortunately, these practices can be applied to any and all of the tasks involved in the deployment of, ongoing maintenance of, and future upgrades to the environment.

Our next post will provide high-level descriptions of each stage in the framework.

We offer these brief definitions to assist in understanding principles in the framework and why they are important. We encourage you to investigate these topics and principles further.

Agile

Agile is an umbrella term for a set of frameworks and practices based on the values and principles expressed in the Manifesto for Agile Software Development and the 12 Principles behind it.

DevSecOps

The definition of DevSecOps is both complex and broad, and changes from organization to organization. For the scope of this post, we define DevSecOps as a personal and organizational mindset defining integrated development (Dev), security (Sec), and operations (Ops) processes for the rapid development, fielding, and operations of software and software-based systems, utilizing automation where feasible, to achieve the desired throughput of new features and capabilities in a secure way. It is built on top of the Agile principles of collaboration and communication and requires organizational culture changes beyond those required to achieve the fundamental Agile values and principles.

Continuous Integration and Continuous Delivery

Continuous integration (CI) is a core exercise when practicing DevSecOps. It occurs when team members are able to submit changes to a central repository at some regular, frequent interval.

Continuous delivery (CD) is the natural evolution of CI. It occurs when changes to a product are automatically validated and subsequently made available to the end users of that product with the confidence that the change operates as expected.