Software Engineering Institute

December 4, 2024—Software containerization has become an expected part of many DevSecOps software development and deployment pipelines. However, external dependencies during the container build process cause successive container images to differ, decreasing trust in containers and possibly obscuring malware insertion. Vessel, an open source, first-of-its-kind tool recently released by the Software Engineering Institute (SEI), spots differences between successive container images and helps sort benign from problematic issues.

Ideally, any developer using the same build environment should be able to reproduce identical builds of a piece of software. But changes in software dependency versions, file-system time stamps, and other external factors can cause software builds to change, leading to unexpected behavior or even failure. In some cases, as in the SolarWinds attack, malware can be injected during the build process. The ability to detect and attribute changes between software builds would increase the trustworthiness of the software supply chain.

Containers provide software applications with all the files they need to run in an isolated environment. Containerization streamlines deployment on different systems and is a standard process in many commercial software DevSecOps pipelines. The practice is also gaining ground in Defense Department software organizations.

But containers are software, so they suffer from the reproducibility challenge. Developers expect container builds to differ, but there has not been an easy way to identify unimportant changes, such as time stamps, and critical ones, such as malware insertion. “You can’t just run a binary comparison of the end result and say that builds are different. They’re always different,” said Kevin Pitstick, a senior software engineer at the SEI and Vessel’s lead developer. “We need an understanding of what differences are benign and which are potentially malicious.”

To help software developers understand those differences, Pitstick and his colleagues started the two-year Vessel project. As their first year concludes, they have released an open-source tool that compares two built container images, reports any differences, and flags many known-benign issues. Unflagged differences could represent malware and other tampering. Vessel’s output can also identify dependencies on changing external resources, such as installs from package managers. With this knowledge, developers can make their software more reproducible, stable, secure, and trustworthy.

These characteristics are especially important for software in the defense and national security domain. Yet software reproducibility has proven difficult in the private sector, where release speed, capability, and production-level security vulnerabilities take priority. As a federally funded research and development center, the SEI’s mission is to fill these kinds of gaps for the government and, ultimately, the larger software community.

Pitstick noted that reproducibility is a big problem space, and no one had yet analyzed reproducibility at the level of container images. He and his team did the groundwork of finding open-source repositories, building images, and figuring out what factors differ and how they manifest.

The team wanted Vessel to do more than generate long lists of differences. Vessel users can compare container images and set up custom rules for categorizing differences as concerning or unimportant.

“Everybody aspires to create software that's reproducible, but to actually get there is a lot of work,” he said. “One of the goals behind the Vessel project is to make a tool that will help developers in this process.”

The SEI team plans to add more rules to Vessel and test it against more use cases. In the meantime, they hope the software community will use Vessel as a foundational model for container reproducibility efforts.

Download Vessel from the SEI’s GitHub page. Learn more about the Vessel project in Pitstick’s 2024 SEI Research Review presentation.