Software Engineering Institute

In a highly regulated environment, container images are built by a pipeline that enforces a variety of constraints, for example, to use the latest version of software dependencies. Existing tools can update top-level dependencies but stop short of updating entire dependency trees. A more complete solution would also update sub-dependencies required during a build. Our solution and the subject of this talk, uses a 'dependency pipeline'. More specifically, this is a series of automation steps which download and neatly package our container image’s dependencies, preparing them for installation in the pipeline of a given highly regulated environment. Using our dependency automations have cut the maintenance times of our developers from hours of effort each week to mere minutes. Additional key benefits of our solution are dependency version conflict avoidance and immediate CVE resolution. A Linux container (LXC) is typically composed of a set of files from custom software builds, downloaded dependencies, and common OS-specific files that are bundled together to deliver some reproducible functionality. Many of these files likely originate from open-source software repositories. Highly regulated environments pose stringent constraints on the functionality of active systems within their bounds such as disallowing the downloading of files from the open Internet. A common requirement for container builds in highly regulated environments is the use of dependencies that are locally stored or downloaded from the open internet using pre-approved package managers and repositories, which typically only contain a small subset of a container build’s needed dependencies. Successful container builds in this environment, are commonly obligated to use a list of required dependencies, which are not available in pre-approved repositories. The list includes for each dependency: a filename, hash value of the file, and an open internet download URL. During the build process, the list of files is downloaded from the provided URLs and placed into a clean local storage build context, within the highly regulated environment, and hash values are generated and compared to ensure file integrity. The validated files are then made available for container builds. Once built, the container enters a long-term maintenance lifecycle phase. In this phase, individual dependency updates are performed to resolve detected CVEs and CCEs, build errors, new version releases, etc... These updates, which are normally done by manually updating individual dependency entries in the required dependencies list, can cause version conflicts. This often occurs with the version update of a single dependency which contains sub-dependencies requiring the older version, or a deprecation of required functionality from the new version. The result is a build failure of unable to locate a compatible version. The current remedy is the non-trivial task of manually identifying and updating versions of multiple individual entries in the required dependencies list which often creates a high volume of cyclical sub-dependency version conflicts requiring hours of effort to analyze and mitigate. Our solution, as described above, resolve these issues.

Rob McCarthy is a DevOps Engineer at the Software Engineering Institute. He works most closely with systems architecture and design surrounding Continuous Integration and Deployment operations. In his spare time Rob enjoys playing Red Team Capture the Flag and spending time with his son, Kade.

Jose Morales is currently a Senior Researcher in the Software Engineering Institute, in the SSD Division, at Carnegie Mellon University. He has conducted research in cyber security since 1998 . He is a co-author of four best paper awards. He has conducted research in DevSecOps since 2016 and has focused on its application to diverse problem sets including highly regulated environments, artificial intelligence, and embedded systems. He graduated with a Ph.D. in Computer Science from Florida International University in 2008. He is co-founder and moderator of the Hispanics in Computing email list. He is a Senior Member of the ACM and IEEE.

Download the graphic recording.