At a recent workshop we hosted, a participant asked why the release frequency was so high in a DevOps environment. When working with significant legacy applications, release may be a once-in-a-year type event, and the prospect of releasing more frequently sends the engineering teams running for the hills. More frequent releases are made possible by properly implementing risk mitigation processes, including automated testing and deployment. With these processes in place, all stakeholders can be confident that frequent releases will be successful.

When testing our software builds, trust becomes a numbers game. What level of certainty do we have that an application will build and deploy successfully if we have only tested the whole process twice? What level of certainty do we have that an application will build and deploy successfully if we have tested that process 600 times? If my application passed 595 of the last 600 tests (with adequate coverage) and deployed to production-like environments, we can be more certain of successful performance than if it has passed one of only two tests.

The frequency that is important in DevOps is not the release-to-production frequency, but the frequency of releasing to production-like environments. Ensuring environment parity through the use of infrastructure as code allows teams to test their deployments consistently and continuously. This frequent testing increases the stakeholders' confidence that the software will deploy reliably. More frequent releases without testing frameworks and automation in place will introduce more chaos into the release cycle, and ultimately more vulnerabilities and problems into production.

Testing should actually test what is critical to the application. Planning, communication, and code reviews help all teams involved identify what testing should be in place to properly mitigate critical risks to the business goals of the application. Mitigating risk gives all team members the freedom and confidence to explore and innovate without fear that they will deploy features that detract from the business value of the application.

DevOps allows for failure in the correct phase. For example, developers, by the nature of their jobs, will introduce instability, and if the instability breaks the application, the developers should be notified immediately. If that instability is instantly pushed to production, then DevOps principles are not being followed. The longer a problem exists before it is fixed increases the cost to fix it. Testing frequently and automatically across all phases of development is vital.

If you cannot fully test the application after every build, test only the parts that are important. Tests can be added as needs are identified by the engineering teams and stakeholders involved. The frequency of the testing will depend on the types of tests that are needed and the resources required to run the tests. In some cases, it is better to make the application based on microservices rather than monolithic. By so doing, you enable testing of the small chunks being changed and the deployment only of those chunks successfully changed, thereby reducing your risk profile.

If an application takes three hours to build, it is probably not feasible to run a full suite of tests on it after every developer commits. Perhaps, however, there is a subset of tests that is run after every commit, and the full test suite can run at night. If fuzz testing is important to the application, but it takes three days to test the application, then run fuzz testing every three days. All these processes should be automated so the engineers (whether the quality assurance team, developers, or operations) can focus on other tasks.

In some environments, more frequent releases to customers are not feasible. Whether release cycles are measured in years or hours, frequent testing, especially testing deployment to production-like environments, increases stakeholder confidence that business value will be delivered. Even if the release is two years down the road, the releases have been tested in an automated fashion on a daily, weekly, or monthly timeframe throughout the lifetime of the project.

Every two weeks, the SEI will publish a new blog post offering guidelines and practical advice for organizations seeking to adopt DevOps in practice. We welcome your feedback on this series, as well as suggestions for future content. Please leave feedback in the comments section below.

Additional Resources

To view the webinar Culture Shock: Unlocking DevOps with Collaboration and Communication with Aaron Volkmann and Todd Waits, please click here.

To view the webinar What DevOps is Not! with Hasan Yasar and C. Aaron Cois, please click here.

To listen to the podcast DevOps--Transform Development and Operations for Fast, Secure Deployments featuring Gene Kim and Julia Allen, please click here.

To read all of the blog posts in our DevOps series, please click here.