When it comes to information technology services that are customer facing, traditional enterprise organizations tend to favor stability over change. According to a Netcraft survey from March of last year, there were 185 million web sites hosted by Windows 2003, an operating system that has been out of support since July 2015 . Many of these servers are still running because of the "if it isn't broken, don't fix it" motto. While reducing software and system churn would seem like the best way to promote stability, it can eventually harm application security and stability. This blog post explores some basic DevOps practices that will improve application security while helping to maintain a stable operating environment.

When an organization's primary product is not software, system uptime is important so the primary product can be sold. For example, if Equifax cannot process credit report requests, they lose an important stream of income, which is akin to an endless line of customers behind a broken cash register. Unfortunately, in an attempt to reduce churn, organizations like Equifax run unsupported server operating systems, outdated web frameworks, or unpatched application servers. Ironically, Equifax was burned by a flaw in Apache Struts web framework, which could have been prevented if the application in question had an automated DevOps pipeline. In many cases, these situations stem from a lack of resources that can be devoted to application upgrades, server migrations, or manual testing.

Although the "if it isn't broken, don't fix it" mentality cannot be changed overnight, the benefits of automated software and system changes can be demonstrated if DevOps practices are slowly adopted by a project or application support workflow. Since stability and change are metrics that can be measured, other projects and application teams will be interested in reducing risk, thus helping to spread the culture of DevOps throughout the organization.

Infrastructure as Code

Although configuration and deployment vary from organization to organization, a system engineer generally configures the base server operating system, leaving the application configuration and deployment to the application team. When an application must be moved to a supported operating system, the server team creates a new server and hands it over to the application team. The application team will then configure the server using a checklist or by trial and error. This process is typically plagued by configuration errors that are discovered by manual testing or, worse, in production.

For example, errors can occur due to differences in the operating system images, edge-case scenarios, newer software versions, or steps that were missed on the configuration checklist. After the testing has been completed, the test environment is then cloned for production usage, or it is moved directly into production. These arduous tasks must be completed for each server migration. DevOps can improve this process by automating the server and application configuration. DevOps also requires both the systems and application teams to work together since the running application is the final product, not just a server and, separately, the application.

To maintain consistent environments, a DevOps practice, known as Infrastructure as Code (IaC), can be used. The concept of IaC is simple; your server configuration is stored in a source code repository with the application source. Automation tools, such as Ansible, Chef, or Puppet, will take the configuration artifact (normally written in a simple markup or scripting language, such as YAML or Ruby) and apply each task to the desired system. These tools are easy to learn and can automate virtually any task.

Your configuration tool of choice can configure your application user accounts, install packages to meet dependencies, adjust the system environment, maintain web server configuration, create folders, or set folder and file permissions. In addition to configuration, automation tools can often bridge the gap between manual application deployment and a full continuous delivery pipeline. These tools can also automatically apply and revert any manual changes by running scheduled system validation checks; however, this is a more advanced process and requires some additional configuration.

By automating a small portion of your environment configuration, you have increased the chances of a successful migration; additionally, the same IaC can be used to migrate your application to the cloud if desired in the future. In addition to repeatable environment configuration, IaC also allows you to track environment changes. Since IaC is stored in a source code repository, you will automatically have a history of all environment changes, dependencies, and dependency versions.

For example, if you were using Apache Struts, you would be able to quickly identify an insecure version of the framework. You would also know when a security risk was introduced via a third-party dependency. These insights will help provide a timeline if security vulnerabilities of your application are identified.

Automated Testing

To ensure compatibility and stability, all migrations and upgrade processes should have a regression test plan. Unfortunately, many brownfield applications lack a test plan or prior knowledge of business functionality. These migrations or upgrades require a new test plan, and numerous hours of labor, which some could construe as wasted time. Additional labor is not always available, so any upgrade, security patch, or migration is put in a backlog, sometimes indefinitely.

Some organizations consider the development of automated tests too costly to create. While automated tests can take a signification amount of time and effort to create, the benefit outweighs the cost. For example, a small series of end-to-end tests can be initially created, and more tests can be added as issues arise. This test suite can be executed before and after any migrations or upgrades. The test suite will identify any known issues, saving manual testing time. Manual testers can focus on testing the more complex features that cannot be quickly automated.

After the test suite coverage nears 100 percent of the application, you can confidently migrate, patch, or otherwise deploy your application. Automated testing need not be limited to application-specific tests; numerous security tests can be run against the application to expose any security vulnerabilities. The scans could attempt cross-script scripting (XSS), SQL injection, server file system traversal, or other security exploits to expose any application vulnerabilities. In addition to vulnerability scans, the test suite can also perform static code analysis and fuzzing to find more uncommon security flaws.

Deployment Automation

Software deployment in many organizations ranges from copying files to a file share to a fully automated deployment pipeline. In most cases, deployment is manual. The original application is backed up, the new code is copied to a location on a server, and perhaps, some file permissions are set or a server process is restarted. While deployment is relatively simple for small applications, it can be very complex if the application touches many servers. The more complex the workflow, however, the easier it is to experience an outage or error due to a missed deployment step.

Deployment automation is the first step in creating a continuous delivery pipeline, which is another DevOps practice. The goal of continuous delivery is the ability to make a code change, build and test the application, and deploy the new build to a user-acceptance testing or production environment. Organizations that are new to DevOps, however, should first focus on creating an automated deployment process.

Deployment automation limits the chance for user error and avoids any vague guesswork when deploying an application. Since automation removes user error from the deployment process, you can be confident in deploying your application for a security patch, bug fix, or to a new environment. There will not be any worries that a particular step is missed; each deployment will follow the same pattern as the last. Likewise, deployment time is reduced. In certain cases, deployment can be reduced from hours down to minutes, depending on the complexity of the system.

Wrapping Up and Looking Ahead

DevOps processes can reduce manual (non-automated) labor, reduce risks by eliminating environmental configuration differences, while increasing confidence of engineering and business partners. Although traditional mindsets are hard to change, DevOps practices can be slowly injected into brownfield applications and support teams. After the benefits are realized, the integration of DevOps can change the culture of an organization by reducing risks and increasing team collaboration.

This post provides DevOps solutions for common problems that plague the software development lifecycle. In the future, this series will take a deeper dive into the following topics:

• dependency management
• infrastructure as code
• deployment automation
• automated testing

Additional Resources

View our latest SEI Podcast, How Risk Management Fits into Agile DevOps and Government.