Category: DevOps

We often discuss how important it is to incorporate security into all parts of the DevOps software development lifecycle (SDLC). For example, my post Security...Security Everywhere discusses what types of security can be incorporated into the different phases of the SDLC. However, incorporating security is often hard, due to part to the fact that most automated security testing tools are only available in a couple of places in the SDLC, primarily the continuous integration (CI) server. There is an opportunity for lots of testing without much additional overhead. This opportunity presents itself when developers push their code to a central code repository, specifically git repositories. Using git hooks, developers can write tests for their code and run them when code is committed and pushed to the repository. These tests will actually prevent developers from committing and pushing their code if they contain security flaws. In this blog post, I will introduce and present a demonstration of Overcommit, an open-source tool that manages git hooks.

Software development project stakeholders can often be tempted to put security requirements on the back burner when developing software systems. During one particular large-scale software development project I was involved with, which was a distributed system consisting of many components communicating over the network, runtime performance was the most important quality attribute. The engineers brilliantly invented their own lightweight protocol to maximize runtime performance. Once the system was to be transitioned into production operations, it was discovered that encryption and authentication had to be added to comply with the security requirements of the customer's site. These changes resulted in degraded runtime performance and delayed the system's ability to be used in the field. The project went over schedule, and costly rework had to be performed to accommodate the added overhead of secure communications. In this blog post, I will explore rethinking software requirements to maximize security and minimize risk.

Awareness and adoption of DevOps continues to grow. A 2016 DevOps trends report found that DevOps adoption increased from 66 percent in 2015 to 74 percent in 2016

In 2016, visitors to the SEI DevOps Blog were drawn to posts highlighting successful DevOps implementations at Amazon and Netflix, as well as tutorials on Fabric, Ansible, and Docker. This post presents in descending order (with number one at the bottom being the most popular) the five most popular DevOps posts in 2016.

The term "software security" often evokes negative feelings among software developers because it is associated with additional programming effort, uncertainty, and road blocks on fast development and release cycle. To secure software, developers must follow numerous guidelines that, while intended to satisfy some regulation or other, can be very restrictive and hard to understand. As a result, a lot of fear, uncertainty, and doubt can surround software security. This blog post, the first in a series, is based on a keynote I recently delivered at the International Conference on Availability, Reliability, and Security (ARES). In this talk I describe how the Secure DevOps movement attempts to combat the toxic environment surrounding software security by shifting the paradigm from following rules and guidelines to creatively determining solutions for tough security problems.

Vagrant Box Wrangling

By on in

So, you're using Vagrant, and maybe you've even read my earlier post on it, but your Vagrant box doesn't have everything you need. Or maybe it has too much, and you need something simpler. For instance, do you find yourself installing or removing packages or fixing packages to specific versions to get parity with your production platform? Or maybe you need more extensive auditing over your environment, such as when you (or your customer) can't trust a third-party box vendor. Or you need a way to clone a virtual machine for parity with the production environment. What are your options? In this blog post, I will explain what a box file is and how you can have more control over your Vagrant workflow by creating your own box. I will also introduce Packer as a tool to create a Vagrant box, and I will finish with an example for managing Vagrant box versions and distributing updates in a team setting.

Change is hard. When we help teams adopt DevOps processes or more general Agile methodologies, we often encounter initial resistance. When people learn a new tool or process, productivity and enthusiasm consistently dip, which is known as the "implementation dip." This dip should not be feared, however, but embraced. In his book Leading in a Culture of Change, Michael Fullan defines the implementation dip as "a dip in performance and confidence as one encounters an innovation that requires new skills and new understandings."

A shift to DevOps is a shift to constantly changing and improving tools and processes. Without deliberate steps, we could thrust our team into a constant cycle of implementation dip. In this blog post, I present three strategies for limiting the depth and duration of the implementation dip in software development organizations adopting DevOps.

In the ever-changing world of DevOps, where micro-services and distributed architectures are becoming the norm, the need to understand application internal state is growing rapidly. Whitebox monitoring gives you details about the internal state of your application, such as the total number of HTTP requests on your web server or the number of errors logged. In contrast, blackbox testing (e.g., Nagios) allows you to check a system or application (e.g., checking disk space, or pinging a host) to see if a host or service is alive, but does not help you understand how it may have gotten to the current state. Prometheus is an open source whitebox monitoring solution that uses a time-series database to provide scraping, querying, graphing and alerting based on time-series data. This blog post briefly explores the benefits of using Prometheus as a whitebox monitoring tool.

It has been nearly a year since the DevOps blog launched its own platform. In the nearly 12 months since our launch, we have offered guidelines, practical advice, and tutorials to the ever-increasing number of organizations adopting DevOps (up 26 percent since 2011). In the first six months of 2016, an increasing number of blog visitors were drawn to posts highlighting successful DevOps implementations at Amazon and Netflix as well as tutorials on new technologies such as Otto, Fabric, Ansible, and Docker. This post presents in descending order (with #1 being the most popular) the 10 most popular DevOps posts published in the first six months of 2016.

In this DevOps revolution, we are trying to make everything continuous: continuous integration, continuous deployment, continuous monitoring--the list goes on. One term you rarely hear, however, is continuous security, because it is often seen as an afterthought when building and implementing a delivery pipeline. The pipeline I will be discussing has six components: plan, code, build, test, release, and operate. There is also a seventh, less-formal component, which is the iterative nature of the delivery pipeline in a DevOps environment. Security can, and should, be implemented throughout the pipeline. In this blog post, I discuss how security can be implemented in all components, as well as what benefits come from implementing security in each of those components.

DevOps practices can increase the validity of software tests and decrease risk in deploying software changes to production environments. Anytime a software change is deployed to production, there is a risk that the change will break and lead to a service outage. This risk is minimized through rigorous testing of the software in a separate test environment where the change can be safely vetted without affecting normal business operations. Problems can arise, however, when these isolated test environments do not properly mimic the production environment. Sometimes a test environment will have different operating system patches installed, different software dependencies installed, different firewall rules, or even different data in its database. These differences open the door to risk because even if the software change passes testing in the test environment, there is a chance of failure in production because it was never before tested in that context. In this blog post, I explore how the DevOps practices of infrastructure as code and automated test execution through continuous integration increase the effectiveness of software testing, allowing us to create test environments that more closely match production.

A few years ago, my team took the task of designing and writing a new (and fairly large) web application project that required us to work collaboratively on features, deploy to unfamiliar environments, and work with other teams to complete those deployments. Does this sound like DevOps yet? Our task was to make these deployments happen with limited resources; however, we didn't want to sacrifice environment parity or automation, knowing that these would help our team lower risk for the project and maintain a better handle on the security of our process. The idea of using Chef, a leading suite of platform-independent infrastructure management and provisioning tools, came to mind; however the work that would have been required to implement the full Chef ecosystem was not in scope for the project. We realized, however, that we were already using Vagrant to provision our local environments in the same way we wanted to provision our remote environments. That is, our Vagrant-based workflow was applying Infrastructure as Code concepts and provisioning with just a single component of Chef, rather than depending on a larger suite of Chef tools. To solve our remote deployment problem, we furnished a solution that allowed us to maintain environment parity by reusing all of the existing Chef configuration while sharing it with Vagrant and keeping the deployment for any size system to a single, automatable line. In this blog post, I will be talking about the transformation of a vanilla Vagrant setup to a stable and testable Infrastructure as Code solution.

I am often asked how to help DevOps organizations improve their software and system security by integrating security testing into their new and expanding continuous integration (CI) environment. The first thing I say is, "It is great that you are treating security testing as important a task as other software tests." Security testing is often overlooked or simply manually done at the end of a software release cycle, if at all. When I ask them, "What type of security testing do you currently do?" I often hear excuses about the lack of time, funding, or planning as the reason they do not currently perform any security testing at all. DevOps organizations typically do not have a security testing plan in place, have not really given it much thought, and hope that CI alone can help make their software more secure. As this blog post will detail, CI certainly can help improve your application security if you are already automating security testing, but you must first have a security plan in place.

Traditionally, DevOps practitioners think of business value as simply measuring the difference between money earned and money spent. In that line of thinking, security is often relegated to a secondary goal because it fails to directly drive revenue. The misguided goal is to deliver functionality at all costs, even if it compromises the integrity of the system or data. As Rob Joyce, head of the National Security Agency's Tailored Access Operations group, mentions in his talk at Usenix Enigma conference: "Don't assume a crack is too small to be noticed, or too small to be exploited... We need that first crack, that first seam. And we're going to look and look and look for that esoteric kind of edge case to break open and crack in." In this blog post, I present two concepts: malicious user stories and rejection criteria that can be used in DevOps to secure systems.

DevOps practitioners often omit security testing when building their DevOps pipelines because security is often linked with slow-moving business units and outdated policies. These characteristics conflict with the overall goal of DevOps, which is to improve the software delivery process. However, security plays an important role in the software development lifecycle and must be addressed in all applications. Incorporating security into different stages of the DevOps pipeline will not only start to automate security, but will allow your security process to become traceable and easily repeatable. For instance, static code analysis could be done on each code commit, and penetration testing could be completed as part of the deployment phase. In this blog post, I will present two common tools that can be used during deployment that allow for automated security tests: Gauntlt and OWASP Zed Attack Proxy (ZAP).

By Hasan Yasar
Technical Manager
Cyber Engineering Solutions Group

In August 2015, the DevOps blog launched its own platform. The blog offers guidelines, practical advice, and tutorials to the ever-increasing number of organizations adopting DevOps (up 26 percent since 2011). According to recent research, those organizations ship code 30 times faster. Despite the obvious benefits of DevOps, many organizations hesitate to embrace it, which requires a shifting mindset--and cultural and technical requirements--that prove challenging in siloed organizations. Given these barriers, posts by CERT researchers have focused on case studies of successful DevOps implementations at Amazon and Netflix, as well as tutorials on popular DevOps technologies, such as Fabric, Ansible, and Docker. This post presents the 10 most popular DevOps posts published in 2015 in ascending order.

By Tim Palko
Senior Member of the Technical Staff
CERT Cyber Security Solutions Directorate

In the realm of DevOps, automation often takes the spotlight, but nothing is more ubiquitous than the monitoring. There is value to increased awareness during each stage of the delivery pipeline. However, perhaps more than any other aspect of DevOps, the act of monitoring raises the question, "Yes, but what do we monitor?" There are numerous aspects of a project you may want to keep an eye on and dozens of tools from which to choose. This blog post explores what DevOps monitoring means and how it can be applied effectively.

By Aaron Volkmann
Senior Research Engineer
CERT Division

The DevOps philosophy prescribes an increase in communication and collaboration between software development and operations teams to realize better outcomes in software development and delivery endeavors. In addition to bringing development and operations closer together, information security teams should be similarly integrated into DevOps-practicing teams. An automated way of performing complete software security assessments during continuous integration (CI) and continuous delivery (CD) does not exist yet, but this blog post describes how we can apply DevOps principles to application security assessments and integrate the results of those activities with CI/CD.

By Aaron Volkmann
Senior Research Engineer
CERT Division

You will be hard pressed to find a DevOps software development shop that doesn't employ Vagrant to provision their local software development environments during their development phase. In this blog post, I introduce a tool called Otto, by Hashicorp, the makers of Vagrant.

By Todd Waits
Project Lead
CERT Division

DevOps principles focus on helping teams and organizations deliver business value as quickly and consistently as possible. While the principles advocate for improving the coordination between development and operational teams, they can be adapted for any number of domains. The key components of DevOps we want to emulate across other domains are:

  • collaboration between project team roles
  • infrastructure as code
  • automation of tasks, processes, and workflows
  • monitoring of applications and infrastructure

In this blog post, I explore how to apply DevOps to the incident response domain. In the same way that advances in methodologies surrounding software development were gleaned from Toyota's manufacturing processes, we can apply lessons learned from DevOps across domains.

Aaron Volkmann
Senior Research Engineer
CERT Division

In response to several corporate scandals, such as Enron, Worldcom, and Tyco, in the early 2000s congress enacted the Sarbanes-Oxley (SOX) act. The SOX act requires publicly traded companies to maintain a series of internal controls to assure their financial information is being reported properly to investors. In an IT organization, one of the main tenets of SOX compliance is making sure no single employee can unilaterally deploy a software code change into production. DevOps automation techniques and technologies, such as continuous integration (CI), continuous delivery (CD), and infrastructure as code (IaC), can appear on the surface to throw a shop out of SOX compliance. This blog post examines how DevOps automation can help organizations not only stay compliant, but actually increase their compliance.

DevOps for Contractors

By on in

The challenges of DevOps--a cultural change, learning new technologies, and making a big-picture impact for a software project team--are possibly even more challenging in contract work. In this blog post, I'll expand on some of my past experiences as a contract software developer and discuss, in retrospect, how DevOps could have worked in different scenarios.

Formal documentation (such as source code documentation, system requirements and design documentation, or documentation for various user types) is often completely ignored by development teams; applying DevOps processes and philosophies to documentation can help alleviate this problem. Software documentation tends to fall into several categories: code, requirement, design, system, and user documentation. One reason documentation is often ignored is that standard documentation tools and processes create an obstacle for development teams since the tools and processes do not fit in well with the suite of tools development teams rely on, such as version control, issue trackers, wikis, and source code. As a consequence of this mismatch, slow the velocity of development teams. This blog post explores three primary challenges to documentation--process, documenting source code, and system documentation--and explains how DevOps-based documentation allows all stakeholders to access a common, trusted source of information for project details.

Since beginning our DevOps blog in November, and participating in webinars and conferences, we have received many questions that span the various facets of DevOps, including change management, security, and methodologies. This post will address some of the most frequently asked questions.

In late 2014, the SEI blog introduced a biweekly series of blog posts offering guidelines, practical advice, and tutorials for organizations seeking to adopt DevOps. These posts are aimed at the ever-increasing number of organizations adopting DevOps (up 26 percent since 2011). According to recent research, those organizations ship code 30 times faster. Despite the obvious benefits of DevOps, many organizations hesitate to embrace DevOps, which requires a shifting mindset and cultural and technical requirements that prove challenging in siloed organizations. Given these barriers, posts by CERT researchers have focused on case studies of successful DevOps implementations at Amazon and Netflix, as well as tutorials on popular DevOps technologies such as Fabric, Ansible, and Docker. This post presents the 10 most popular DevOps posts (based on number of visits) over the last six months.

Container-based virtualization platforms provide a means to run multiple applications in separate instances. Container technologies can provide significant benefits to DevOps, including increased scalability, resource efficiency, and resiliency. Unless containers are decoupled from the host system, however, there will be the potential for security problems. Until that decoupling happens, this blog posting describes why administrators should keep a close eye on the privilege levels given to applications running within the containers and to users accessing the host system.

At a recent workshop we hosted, a participant asked why the release frequency was so high in a DevOps environment. When working with significant legacy applications, release may be a once-in-a-year type event, and the prospect of releasing more frequently sends the engineering teams running for the hills. More frequent releases are made possible by properly implementing risk mitigation processes, including automated testing and deployment. With these processes in place, all stakeholders can be confident that frequent releases will be successful.

This post is the latest installment in a series aimed at helping organizations adopt DevOps.
Some say that DevOps is a method; others say it is a movement, a philosophy, or even a strategy. There are many ways to define DevOps, but everybody agrees on its basic goal: to bring together development and operations to reduce risk, liability, and time-to-market, while increasing operational awareness. Long before DevOps was a word, though, its growth could be tracked in the automation tooling, culture shifts, and iterative development models (such as Agile) that have been emerging since the early 1970s.

The federal government continues to search for better ways to leverage the latest technology trends and increase efficiency of developing and acquiring new products or obtaining services under constrained budgets. DevOps is gaining more traction in many federal organizations, such as U.S. Citizenship and Immigration Services (USCIS), the Environmental Protection Agency (EPA), and the General Services Administration (GSA). These and other government agencies face challenges, however, when implementing DevOps with Agile methods and employing DevOps practices in every phase of the project lifecycle, including acquisition, development, testing, and deployment. A common mistake when implementing DevOps is trying to buy a finished product or an automated toolset, rather than considering its methods and the critical elements required for successful adoption within the organization. As described in previous posts on this blog, DevOps is an extension of Agile methods that requires all the knowledge and skills necessary to take a project from inception through sustainment and also contain project stakeholders within a dedicated team.

DevOps can be succinctly defined as a mindset of molding your process and organizational structures to promote

  • business value
  • software quality attributes most important to your organization
  • continuous improvement

As I have discussed in previous posts on DevOps at Amazon and software quality in DevOps, while DevOps is often approached through practices such as Agile development, automation, and continuous delivery, the spirit of DevOps can be applied in many ways. In this blog post, I am going to look at another seminal case study of DevOps thinking applied in a somewhat out-of-the-box way: Netflix.

This post is the latest installment in a series aimed at helping organizations adopt DevOps.

Tools used in DevOps environments such as continuous integration and continuous deployment speed up the process of pushing code to production. Often this means continuous deployment cycles that could result in multiple deployments per day. Traditional security testing, which often requires manually running multiple tests in different tools, does not keep pace with this rapid schedule. This blog post introduces a tool called Gauntlt, which attempts to remedy this issue.

When Agile software development models were first envisioned, a core tenet was to iterate more quickly on software changes and determine the correct path via exploration--essentially, striving to "fail fast" and iterate to correctness as a fundamental project goal. The reason for this process was a belief that developers lacked the necessary information to correctly define long-term project requirements at the onset of a project, due to an inadequate understanding of the customer and an inability to anticipate a customer's evolving needs. Recent research supports this reasoning by continuing to highlight disconnects between planning, design, and implementation in the software development lifecycle. This blog post highlights continuous integration to avoid disconnects and mitigate risk in software development projects.

"Software security" often evokes negative feelings among software developers since this term is associated with additional programming effort and uncertainty. To secure software, developers must follow a lot of guidelines that, while intended to satisfy some regulation or other, can be very restricting and hard to understand. As a result a lot of fear, uncertainty, and doubt can surround software security. This blog posting describes how the Rugged Software movement attempts to combat the toxic environment surrounding software security by shifting the paradigm from following rules and guidelines to creatively determining solutions for tough security problems.

The workflow of deploying code is almost as old as code itself. There are many use cases associated with the deployment process, including evaluating resource requirements, designing a production system, provisioning and configuring production servers, and pushing code to name a few. In this blog post I focus on a use case for configuring a remote server with the packages and software necessary to execute your code.

In a computing system, a context switch occurs when an operating system stores the state of an application thread before stopping the thread and restoring the state of a different (previously stopped) thread so its execution can resume. The overhead incurred by a context switch managing the process of storing and restoring state negatively impacts operating system and application performance. This blog post describes how DevOps ameliorates the negative impacts that "context switching" between projects can have on a software engineering team's performance.

The DevOps movement is clearly taking the IT world by storm. Technical feats, such as continuous integration (CI), comprehensive automated testing, and continuous delivery (CD) that at one time could only be mastered by hip, trendy startups incapable of failure, are now being successfully performed by traditional enterprises who have a long history of IT operations and are still relying on legacy technologies (the former type of enterprises are known in the DevOps community as "unicorns," the latter as "horses"). In this post, I explore the experience of a fictional horse, Derrick and Anderson (D&A) Lumber, Inc., a company that hit some bumps in the road on its way to DevOps. As D&A finds out, a DevOps transformation is not a product that can be purchased from the outside, but rather a competency that must be grown from within.

When building and delivering software, DevOps practices, such as automated testing, continuous integration, and continuous delivery, allow organizations to move more quickly by speeding the delivery of quality software features, that increase business value. Infrastructure automation tools, such as Chef, Puppet, and Ansible, allow the application of these practices to compute nodes through server provisioning using software scripts. These scripts are first-class software artifacts that benefit from source code version control, automated testing, continuous integration, and continuous delivery.

In the post What is DevOps?, we define one of the benefits of DevOps as "collaboration between project team roles." Conversations between team members and the platform on which communication occurs can have a profound impact on that collaboration. Poor or unused communication tools lead to miscommunication, redundant efforts, or faulty implementations. On the other hand, communication tools integrated with the development and operational infrastructures can speed up the delivery of business value to the organization. How a team structures the very infrastructure on which they communicate will directly impact their effectiveness as a team. ChatOps is a branch of DevOps focusing on the communications within the DevOps team. The ChatOps space encompasses the communication and collaboration tools within the team: notifications, chat servers, bots, issue tracking systems, etc.

When Agile software development models were first envisioned, a core tenet was to iterate more quickly on software changes and determine the correct path via exploration--essentially, striving to "fail fast" and iterate to correctness as a fundamental project goal. The reason for this process was a belief that developers lacked the necessary information to correctly define long-term project requirements at the onset of a project, due to an inadequate understanding of the customer and an inability to anticipate a customer's evolving needs.

Development with Docker

By on in

In our last post, DevOps and Docker, I introduced Docker as a tool to develop and deploy software applications in a controlled, isolated, flexible, and highly portable infrastructure. In this post, I am going to show you how easy it is to get started with Docker. I will dive in and demonstrate how to use Docker containers in a common software development environment by launching a database container (MongoDB), a web service container (a Python Bottle app), and configuring them to communicate forming a functional multi-container application. If you haven't learned the basics of Docker yet, you should go ahead and try out their official tutorial here before continuing.

DevOps and Docker

By on in

Docker is quite the buzz in the DevOps community these days, and for good reason. Docker containers provide the tools to develop and deploy software applications in a controlled, isolated, flexible, highly portable infrastructure. Docker offers substantial benefits to scalability, resource efficiency, and resiliency, as we'll demonstrate in this posting and upcoming postings in the DevOps blog.

On the surface, DevOps sounds great. Automation, collaboration, efficiency--all things you want for your team and organization. But where do you begin? DevOps promises high return on investment in exchange for a significant shift in culture, process, and technology. Substantially changing any one of those things in an established organization can feel like a superhuman feat. So, how can you start your organization on the path to DevOps without compromising your existing business goals and trajectories?

Environment parity is the ideal state where the various environments in which code is executed behave equivalently. The lack of environment parity is one of the more frustrating and tenacious aspects of software development. Deployments and development both fall victim to this pitfall too often, reducing stability, predictability, and productivity. When parity is not achieved, environments behave differently, which makes troubleshooting hard and can make collaboration seem impossible. This lack of parity is a burden for too many developers and operational staff. Looking back on almost every problem I have seen in new production deployments, I find it hard to think of one issue that wasn't due in some part to lack of parity. For developers, this pain is felt when integrating and testing code.