To maintain a competitive edge, software organizations should be early adopters of innovation. To achieve this edge, organizations from Flickr and IBM to small tech startups are increasingly adopting an environment of deep collaboration between development and operations (DevOps) teams and technologies, which historically have been two disjointed groups responsible for information technology development. "The value of DevOps can be illustrated as an innovation and delivery lifecycle, with a continuous feedback loop to learn and respond to customer needs," Ashok Reddy writes in the technical white paper, DevOps: The IBM Approach.

Beyond innovation and delivery, DevOps provides a means for automating repetitive tasks within the software development lifecycle (SDLC), such as software builds, testing, and deployments, allowing them to occur more naturally and frequently throughout the SDLC. This blog post, the second in our series, presents a generalized model for automated DevOps and describes the significant potential advantages for a modern software development team.

I oversee a software engineering team that works within CERT and focuses on research and development of solutions to cybersecurity challenges. Our engineers design and implement software solutions that solve challenging problems for federal agencies, law enforcement, defense intelligence organizations, and industry by leveraging cutting-edge academic research and emerging technologies.

The environment and manner in which software engineering teams (not just my own) write code are constantly evolving. Until recently, most software engineers operated in siloed environments, with developers in one silo and an operations staff of IT professionals who maintained systems and software deployments in another. Members of the operations staff were often uninvolved in development processes, sometimes even unfamiliar with technologies used in development, yet expected to handle deployment, release testing, maintenance, and support of software applications produced by development teams.

Likewise, development teams were unable to design or develop software to be effectively released, tested, and maintained by operations teams. The problem, of course, is that they had no understanding of the needs or processes inherent to their operations teams. This divergence led to wasted time and effort, as well as heavy amounts of risk for software products and deployments.

The concept of DevOps initially evolved in 2009 as an effort to remove barriers between the development and operations teams in software development. At the SEI, we are taking that concept and pushing it forward, alongside many others working in the software industry today.

As a federally-funded research and development center (FFRDC), the SEI must maintain high standards of efficiency, security, and functionality. Our team in particular develops tools and technologies to help federal agencies assess cybersecurity risks, manage secure systems, and investigate increasingly complex cyber attacks and crimes. Cybersecurity is often misunderstood or even ignored as new systems are designed and developed, falling out of view to more high profile quality requirements, such as availability or correctness of software systems. This prioritization necessarily requires addressing security after the primary design phase, or adding security controls after large portions of development have already occurred. Due to CERT's responsibility to our sponsors and the community, security is consistently a first-tier concern, addressed as an early and fundamental requirement for any system developed by our team. In addition to being a defining factor of our software development methodology, this posture heavily influences our approach to DevOps, weaving security considerations into every facet of our software development operation.

As mentioned in my introduction, forward-thinking approaches to process--including heavily automated DevOps techniques--allow us to systematically implement, maintain, and monitor status and quality standards for each of our projects. One large component of the current DevOps movement is Release Automation, which automates the build and deployment cycles of a software project. In Agile software-development scenarios, this process has two benefits:

• automation of final software deployments
• enabling the automation of continuous incremental deployments, triggered numerous times daily as changes are made to the software system

This process ensures that the entire development and management team is aware of the up-to-the-minute state of the software, including its test status and ability to be deployed to the expected runtime environment, continually throughout development. This process also enables highly confident development, and a great deal of certainty that when the time comes, the software can be successfully transitioned to an operations team for deployment and maintenance. This certainty is based on the fact that the entire test and deployment process has been automated and performed countless times already throughout the project lifecycle.

Continuous deployment means no surprises for operations staff, which translates into predictable, low-risk releases of provably high-quality software. Many players in the modern software industry share this vision. "When it comes to ensuring quality, release automation is another major asset for development teams. It automates the complex workflows required for agile software movement between development, test and production environments by removing manual interaction so humans don't introduce avoidable errors," Ruston Vickers wrote in a March 5 blog post on Wired's Innovation Insights blog. That same post cites a study in which more than half of respondents (52 percent) "identified the ability to simplify, standardize and execute application releases with [fewer] errors as the main benefit of release automation." In my view, release automation allows teams to perform build, deployment, and testing activities thoroughly and continually, dramatically increasing the confidence in the state of the software and all processes surrounding release and transition.

In a traditional, siloed environment, teams of developers deploy software manually, on a periodic basis (such as once per quarter, anum, or project cycle) or as necessitated by a new product release. In contrast, for a team leveraging automated DevOps, continuous integration (the process of building and testing a software project continuously, each time new changes are made) is in place and, as part of the process, the project is continuously deployed to an integration environment for testing and review. Some highly mature DevOps operations practice Continuous Deployment, an uninterrupted process that actually deploys live software to production environment.

Automating a task requires a substantial level of understanding by all involved of the processes, technologies, and complexities of deployments of the software in question. This requisite increase in understanding makes it likely that teams moving towards automated DevOps and continuous deployment capabilities also see a positive impact in the reliability, testability, security, and other quality attributes of their software. To achieve automation at this level, the team must study and deeply consider the needs of the entire project, from inception to deployment, which will result in a superior product due to increased focus on implementation details and operational realities.

To achieve the level of automation described, a number of autonomous systems must be in place, such as source control, build and deployment systems, an integration environment, and other systems to channel data and communications throughout DevOps processes. As the image below illustrates, DevOps involves a number of systems operating in concert and communicating seamlessly with other systems and with humans, to detect changes, perform the necessary autonomous functions, and notify team members of status and results. These structured interactions ensure that developers, quality-assurance staff, managers, and even external stakeholders receive continuous, real-time information on the status of the project, which is hugely beneficial, especially in Agile environments. Consistent with the Agile Manifesto: more information, more often, will lead to better project outcomes.

Systems of an Automated DevOps Environment

The remainder of this blog post explores our generalized model for automated DevOps, identifying the systems required to support an automated DevOps process (as shown in the illustration above):

• Source control. Software developers need to safely store their code and keep track of source-code history and versions. For this reason alone, source control is of critical importance. Moreover, in an automated DevOps system, the chosen version control system (VCS) (e.g., Git and Subversion etc.) becomes the system that defines when changes have been made to the software project, triggering the rest of the automated DevOps build, test, and deploy activities.

• Issue tracking system. An issue tracking system allows everyone involved to track current issues, estimates, and deadlines. Communication between this system and systems for code review and source control can allow invaluable traceability between code changes and project goals.

• Build system. The build system supports continuous integration by building the software, running unit and integration tests, deploying to the integration environment, and performing any other automated checks defined for new versions of the software. This system must detect changes in source-code files from the source-control system, seamlessly communicate with and push data to the integration environment, and notify team members of status at all times.

• Monitoring system. Monitoring systems continuously track all autonomous systems within the DevOps environment, notifying necessary maintenance staff if a system failure occurs. This monitoring requires communication with all autonomous systems used by the DevOps process. As a manager of a software engineering team, I don't want to spend time debugging issues in my infrastructure. I want my developers to be able to continue coding even if a server fails. If a failure occurs, I want appropriate staff notified immediately so the issue can be fixed as quickly as possible without derailing my project.

• Communications system. The constant exchange of information is important. Our team uses email, wikis, and a real-time chat system to enable continuous communication among all members of the project team. Moreover, communications systems are used to allow our automated systems to communicate with humans involved in the project through channels that are already part of their workflow. On my own team, I don't have to check a dashboard to see that my latest commit broke the build; the build server will email and send me and instant message to alert me, in real time. The communications system will also alert my team lead, so everyone is in the loop and knows to fix the issue immediately so that no other developer's work is inhibited.

• Integration environment. The integration environment hosts all of the virtual machines that make up our DevOps environment. The integration environment houses machines running all of the DevOps services listed here and provides servers for continual deployment of software projects. Once a project is created, it has deployment servers, so, at any time, stakeholders can visit those servers and see the current state of working software achieved by the project team.

• Code review system. To ensure software quality, every line of code must be reviewed by a seasoned developer. The practice of reviewing code also accelerates career growth and learning. Unfortunately, it is hard to find time for code review in a busy schedule. Our automated code review system therefore detects new code committed to source control and automatically assigns it to be reviewed by a senior developer. With automatic management of code reviews, we ensure that no piece of code fails to have multiple developers review it before it ends up in the final version of a software product.

• Documentation system. Regrettably, documentation often remains an afterthought in production software projects. To ensure that documentation is written throughout the project, when developers are most conscious of technical details, we have developed an automated system that allows developers to write documentation easily, along with source code.
  
  All our documentation is written in Markdown, a simple plain-text format that can be easily edited in an integrated development environment (IDE) and committed to source control repositories along with source code. Once the documentation files are committed, an automated system builds robust documentation artifacts in a variety of formats (such as HTML, PDF, and Microsoft Word) for consumption by project managers and stakeholders. This process operates through the build system already monitoring source code for changes and has proven comfortable for developers by not requiring them to change tools to generate comprehensive project documents.

As I see it, all the processes described above--from source control to documentation--require some system to perform them in every software project. Of course, before automated DevOps, many of these processes were performed manually, especially manual testing and deployments. Even now, generation and formatting of system documentation or management of code reviews are still manual processes in most organizations. My goal with this post was to define all tasks necessary throughout the SDLC, and to promote the automation of as many of them as possible, thus freeing up developers to do what they do best: design and write amazing code.

Looking Ahead

While this post presented a generalized model for DevOps practices, future posts in this series will present the following topics:

• advanced DevOps automation
• DevOps system integration
• continuous integration
• continuous deployment
• automated software deployment environment configuration

We welcome your feedback on this series, and what DevOps topics would be of interest to you. Please leave feedback in the comments section below.

Additional Resources

To listen to the podcast, DevOps--Transform Development and Operations for Fast, Secure Deployments, featuring Gene Kim and Julia Allen, please visit https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=58525.

To view the August 2011 edition of the Cutter IT Journal, which was dedicated to DevOps, please visit https://www.cutter.com/offer/devops-software-revolution-making-0.

Additional resources include the following sites:

https://devops.com/

http://dev2ops.org/

http://devopscafe.org/

https://www.evolven.com/blog/devops-developments.html

https://www.ibm.com/developerworks/library/d-develop-reliable-software-devops/index.html?ca=dat-