DevOps practices can increase the validity of software tests and decrease risk in deploying software changes to production environments. Anytime a software change is deployed to production, there is a risk that the change will break and lead to a service outage. This risk is minimized through rigorous testing of the software in a separate test environment where the change can be safely vetted without affecting normal business operations. Problems can arise, however, when these isolated test environments do not properly mimic the production environment. Sometimes a test environment will have different operating system patches installed, different software dependencies installed, different firewall rules, or even different data in its database. These differences open the door to risk because even if the software change passes testing in the test environment, there is a chance of failure in production because it was never before tested in that context. In this blog post, I explore how the DevOps practices of infrastructure as code and automated test execution through continuous integration increase the effectiveness of software testing, allowing us to create test environments that more closely match production.

Infrastructure as Code to the Rescue

Infrastructure as code (IaC) tools such as Vagrant, Otto, Chef, and Puppet automate server provisioning, allowing software development and operations teams to collaborate on scripts that provision servers. The effect is three-fold:

• Increased transparency of server configurations among teams.
  IaC scripts are first-class software artifacts that live in source control alongside the application software they support. They provide a common language to define and describe the way a server is configured, making it possible for the security, development, quality assurance, and operations teams to have full visibility of these configurations and be able to effectively collaborate on how they should evolve.
• A server cookie cutter is created. These tools can create exact replicas of a server, assuring that test and production environments have the same configuration and software installed on them. This assurance increases the likelihood that any testing done in the test environment will behave the same way in the production environment.
• Automation. In the past, a lot of human labor would go into manually provisioning servers. This made it so that once a test server was created, operations teams were incentivized to maintain them so they would not have to be built from scratch again. The test servers may be used to test new versions of software dependencies or operating system patches. The test servers can easily get out of sync with production. Through automated server provisioning using IaC tools, it is now economically feasible to throw away old server instances and replace them with newly minted instances that have identical configurations.

Regulatory Hurdles to Testing Against Real Data

Government and industry regulations such as the Sarbanes-Oxley Act (SOX) of 2002 and the Health Insurance Portability and Accountability Act (HIPAA) of 1996 may restrict developers and quality assurance engineers from accessing production data. Restrictions like these could prevent teams from replicating a production database to a test environment, making it impossible to test software changes against this data outside of production. Luckily, there is a DevOps practice that can relieve teams from this restriction.

With automated testing, organizations can test software on restricted data without disclosing the data to the developers or quality assurance engineers. A continuous integration (CI) system typically serves as the platform from which automated tests are executed. The results of these tests can then be transmitted to the developers through a variety means such as emails, chat bots, or web-based dashboards. A possible workflow would work as follows:

1. Replicate production data to a test database.
2. Deploy software to test server.
3. Execute Tests.
4. Notify Developers of results.

Wrapping Up and Looking Ahead

The DevOps practices of IaC and automated test execution through CI increase the effectiveness of software testing and allow testing against environments that more closely match production.

I welcome your feedback and questions in the comments section below.

Additional Resources

View the webinar DevOps Panel Discussion featuring Kevin Fall, Hasan Yasar, and Joseph D. Yankel.

View the webinar Culture Shock: Unlocking DevOps with Collaboration and Communication with Aaron Volkmann and Todd Waits.

View the webinar What DevOps is Not! with Hasan Yasar and C. Aaron Cois.

Listen to the podcast DevOps: Transform Development and Operations for Fast, Secure Deployments featuring Gene Kim and Julia Allen.

Read all of the blog posts in our DevOps series.