By Hasan Yasar

Technical Manager

CERT Secure Lifecycle Solutions Group

This blog post is co-authored by Suzanne Miller.

DevOps is a set of development practices that emphasizes collaboration, communication, and automation throughout the application lifecycle. In DevOps, all stakeholders--including IT operations staff, testers, developers, customers, and security personnel--are embedded from the inception of the project to its end. This blog post describes SEI research and customer engagements aimed at applying DevOps practices that are typically used at the end of the lifecycle to automate governance at the beginning of the development timeline.

DevOps works best in organizations that attempt to follow the Agile tenets and principles found in the Agile Manifesto. Such organizations value iterative, incremental progress and strong collaboration with users and customers. Development cycles in Agile organizations are based on short (1- to 3-week) iterations and small teams that focus on delivering working software to engaged users who can provide ongoing feedback as the solution evolves. This process is compatible with DevOps, which involves the crossing of organizational boundaries and collaborating with stakeholders outside of the development organization.

In its role as a DoD federally funded research and development center, the SEI focuses on how to improve and streamline acquisition practices for software-reliant systems; specifically, we are researching how Agile and DevOps principles, which were originally developed for industry, can be applied to speed introduction of new technology in government settings.

Why the Government Needs Agile and DevOp

As of 2012, the average time from concept to deployment for major DoD IT system developments was 84 months, or 7 years. A February, 2017 report by the Government Accounting Office (GAO) stated that the DoD "pays more than anticipated, can buy less than expected, and, in some cases, delivers less capability to the warfighter." A 2014 report from the Congressional Research Service (CRS) cited criticisms by defense analysts that Pentagon cost overruns and schedule delays "have a debilitating effect on the nation's military and threaten America's technological advantage and military capabilities." The CRS report also noted the Defense Department's efforts to reform its acquisition process "have failed to rein in cost and schedule growth." The U.S. government can no longer afford to develop and deploy new technology at such a slow pace. Significant delays occur not only toward the end of the development lifecycle, but also near the beginning. For example, in many government contexts, approval of a budget request, which precedes an actual development program, can take as long as two years.

Consequently, reducing the time from concept to capability is becoming increasingly critical. Cyber threats are emerging more and more quickly, and defenders of systems must work faster and faster to keep up and respond. The recent Equifax breach is an example of what can happen when defenders don't act at the pace of their adversaries. A patch was available several months before it was applied; had the patch been applied earlier, the damage caused by the breach could have been greatly reduced.

Current acquisition practice, however, makes this type of agility hard to achieve. In current practice, once a technical baseline has been approved, any change to a system requirement necessitates a review by a change-control board that analyzes it, approves, disapproves, or requests more information. The board then forwards the change request to development (usually an industry contractor). From there, the changed product goes to testing, back to the board for another approval, and then into the test suite for further testing. If the product is an airframe, further airworthiness checks are be required. Clearly this pattern is rife with governance bottlenecks.

SEI research aims to understand the extent to which the overall governance process can be improved, especially through the application of principles (and possibly tools) typically associated with DevOps-informed governance at the end of the lifecycle. More specifically we try to identify development practices and platforms that might enable the automation of the governance process.

DevOps practices precisely define the criteria people use when they approve changes and permit development to proceed. Though challenging, defining these criteria greatly speeds development. For instance, we know of one example in the Air Force in which an eight-month airworthiness certification was required whenever a change was made to an iPad that would be fielded to an airframe (because it is on the airplane, the iPad is subject to airworthiness re-certification). By applying DevOps principles to define approval criteria, this process was reduced to less than a week by

1. shadowing the operational test and airworthiness personnel,
2. observing the manual tests they conducted, automating those tests, and
3. conducting follow-up sessions with them in which they asked, "What else needs to happen for this automated test to replace what you're doing manually?"

If the answer to the third step was "nothing," they agreed to allow use of this test as a proxy for the manual process, which constituted a significant change to the governance practices.

Applying Commercial Agile and DevOps Practices in Government Settings

The cadences of commercial development practices often differ from those used in government. For instance, in Agile and DevOps commercial practices, large-batch thinking (common to government settings) is replaced by small-batch thinking. The DevOps paradigm is to deploy small increments to get quick feedback, determine where problems exist in the environment, and identify what should be fixed. DevOps includes all the stakeholders involved in a project from beginning to end, including development and operations personnel, program managers, acquisition staff, and security and quality engineers and others as needed.

DevOps is based on four principles:

1. Collaboration among all stakeholders within the same project team
2. Infrastructure as Code, whereby all assets are versioned, scripted, and shared where possible
3. Automation of deployment, testing, provisioning, or any manual process that is prone to human error (continuous integration/delivery/deployment, CICD)
4. Monitoring by means of any metric in the development or operational spaces that can inform priorities, direction, and policy

Many government systems are developed by contractors in conjunction with the government, but development typically takes place outside of government settings. Consequently, a large part of government effort addresses the beginning of the development cycle, when the program is being conceived and formulated, and the end of the cycle, when operational testing is conducted and the system is deployed. In between, the system is developed primarily within the development environment of the contractor. This lack of parity among the environments at the beginning, middle, and end of development is one issue that must be overcome to get consistent operational results.

IT practitioners have learned that DevOps' continuous integration practices make automated testing faster toward the end of the lifecycle, prior to release. We therefore start by looking at the testing and deployment end of the lifecycle, where industry has been most active in applying DevOps. This phase of the lifecycle presents unique challenges in the highly regulated and air-gapped government environment. These challenges must be solved to assure government stakeholders that adopting Agile and DevOps is feasible.

It is during the early stages of the lifecycle, however, where long development cycles most challenge the DoD and government. Streamlining processes at the beginning of the lifecycle by applying Agile and DevOps concepts would shorten the concept phases of a typical acquisition. By addressing both the concept phase (beginning) and testing-deployment phase (end) of the development lifecycle in this way, development time can be reduced and deployment achieved within the window of operational need.

How We Work with Customers: Agile then DevOps

In a typical government or DoD engagement, the SEI helps the organization implement Agile practices to respond more easily in environments of continual change. When customers are ready, we then bring in our DevOps experts to establish an integrated platform on which DevOps principles can be applied. SEI technical teams develop sets of recommended DevOps practices after assessing current processes and environments. These practices can be customized to suit specific program technical needs.

We also teach acquisition personnel how to build trust relationships and achieve productive cross-contract collaboration. We do this, in part, by developing contracting language that enables and supports Agile and DevOps. Automating decisions and making them transparent also helps to build trust, and trust in the integrated development infrastructure reduces vulnerability to political exigencies. For example, we always apply our criteria for approval by means of checklists, every time an approval is needed. If we need to change these criteria, all parties agree on how to change them, but approval is not based on specific individuals. Our mantra is, we run our checklist criteria every time as part of our continuous integration process.

DevOps promotes collective team decisions. Basing decisions on objective criteria reduces variation, eliminates human bottlenecks, and makes progress less dependent on individuals. While some may perceive this increased objectivity as a potential threat to their importance and power, we work to persuade government participants of the value of this objectified approach. We counter the perception that a human is being "replaced by automation" by emphasizing the valuable and worthwhile things personnel could otherwise be doing, such as innovating or experimenting with new designs, tools, or techniques. Our role is to help people see the upside of change.

Changing Mindsets

The Agile-DevOps approach we recommend represents a change of mindset. Infrastructure as code (IaC), for example, makes this change of mindset possible by automating various environments and maintaining parity among both classified and unclassified environments. IaC requires connectivity to the internet to pull from project dependencies that are available in unclassified environments. In classified environments, however, IaC is challenging to implement due to limited internet connectivity or inability to update the dependencies library repository. Conditions that allow automation and environment provisioning, which automatically build from a set of parameters around a piece of code, were not available in classified environments for many years because these environments were restricted to tools authorized specifically for those environments. Many Agile/DevOps tools on which projects depend were developed within the last 10 years and are prone to frequent change as new versions are released. The government, by necessity, is cautious about the safety of such frequent version changes.

Nevertheless, we have made strides in the intelligence community, which faces some of the biggest challenges concerning classified-unclassified automation. For example, the Joint Improvised-Threat Organization (JIDO) and National Geospatial-Intelligence Agency (NGA) are actively involved in DevOps because they are highly motivated to reduce the time from concept to capability. They have also demonstrated their willingness to change certain practices, such as environment parity, authority to operate (ATO), and integrated development pipeline.

Risks, Benefits, and Obstacles to Change

Like many cultures, the DoD and government are risk averse. For an individual in these environments, it is less personally risky to follow long-established approved procedures and fail than it is to fail when doing things differently. This natural tendency makes it hard to persuade people to change.

Another factor working against change is that the Agile-DevOps mindset represents a more technical approach to governance, which is not familiar to many government personnel. Since 1995, when the government began placing more technical risk on contractors and giving contractors responsibility for leading systems integration, the government engineering workforce has been greatly reduced. Consequently, government technical staff often do not have the technical background to feel comfortable with the Agile-DevOps changes we advocate.

The SEI offers training that can help government personnel learn the skills needed to support these new practices. We have also begun developing an instrument to assess organizations' readiness to adopt Agile and DevOps. We would welcome opportunities to pilot this assessment instrument with your organization.

The SEI has ideas and resources that government organizations can use if they are struggling with Agile and DevOps adoption. Contact us to learn more.

Additional Resources

Watch the webinar Practical Considerations in Adopting Agile/Lean in Government Settings by Suzanne Miller and Eileen Wrubel.

Watch the webinar Five Keys to Effective Agile Test Automation for Government Programs by Robert V. Binder and Suzanne Miller.

Watch the webinar Three Secrets to Successful Agile Metrics with Will Hayes.

Watch the webinar Security Practitioner Perspective on DevOps for Building Secure Solutions.

Watch the webinar What DevOps is Not!