At the Applied Systems Group in the SEI’s CERT Division, one of our areas of work focuses on Docker container images that require hardening to remove vulnerabilities. Through our work, we’ve seen stakeholders encountering difficulty with hardening open-source container images, in addition to developing routine processes for vulnerability mitigation for Docker container images.

Using unvetted container images can increase security risks through the introduction of vulnerable software into an organization’s software supply chain. For example, unvetted container images could contain packages with known Common Vulnerabilities and Exposures (CVEs), introducing a potential exploitation vector. As this blog post details, hardening these container images, or selecting verified pre-hardened container images before usage, decreases the risk of introducing vulnerable software.

Hardening containers and container images involves analyzing their current security status and then applying remediations to iteratively improve security. The hardening process is routine and provides iterative security monitoring during the development process. A hardened image provides a secure sandbox for development and execution of open-source software.

The Container Hardening Process

The process of hardening containers and container images starts with a baseline security scan of the unhardened image to create a list of known vulnerabilities. Once this baseline has been made, engineers should analyze the image Dockerfile and replace any non-standard base images with reputable base images from verified publishers. Following the rebasing, engineers, using container image solutions such as Docker, should begin the iterative process to remediate known bugs while scanning for new vulnerabilities.

It's advisable to replace non-standard images used in a Dockerfile because, as we mentioned earlier, images can bring vulnerabilities that can find their way into the end product. This can take the form of vulnerable packages or other dependencies used by the image, but it can also come as a result of unsecure practices in the image itself. To counteract this, rebasing the Dockerfile to use images from Docker verified publishers such as Red Hat allows for more stability in updates and vulnerability mitigation.

These tactics of scanning and substituting images is essentially the beginnings of a software bill of materials (SBOM), and this makes it easier to create and monitor an SBOM. The outputs of each security scan help identify where actions are needed when new vulnerabilities come to light and can be added to the SBOM to maintain its accuracy.

Reducing Potential Vulnerabilities in Container Images

Several other practices can also be employed when rebasing a Dockerfile to further mitigate risks and vulnerabilities. For starters, if a Dockerfile does not employ it already, engineers can implement a multi-stage build process. This allows the build process to install the dependencies needed to build components of the service, while leaving these build dependencies behind in their respective stages. Doing this allows the build process to bring only what is required for the final image into the final build stage. Leaving these runtime-irrelevant dependencies out of the final image can reduce the image’s threat surface.

In the same vein, a more minimal base image can also be beneficial. Building a container to serve a database, for example, is unlikely to need most of the features of a full operating system (OS) base image. Several common base images such as Red Hat Universal Base Image(UBI), Debian, and Rocky Linux offer stripped-down versions leaving fewer superfluous and potentially vulnerable packages and configurations, and also gives the added benefit of reducing the size of the final image. For more on the topic of reducing the size of images, check out this SEI presentation by Kevin Pitstick.

There are a few other tweaks that can also aid in reducing potential vulnerabilities in container images. One of these is to replace the use of the ADD instruction with the COPY instruction wherever possible. ADD and COPY overlap in that they can both be used to move local files around during the image building process. However, ADD has the additional capability of being able to download files referenced by external URLs, as well as unpackage archives. These additional capabilities can be unwanted additions when security is a concern. Containers should also be set to run as a non-root service user when possible, to limit their ability to perform malicious tasks if compromised. Note that by default, Docker containers run as root. By running a container as a non-root service, the principle of least privilege is followed. It’s also a good practice to define the service user early on in a Dockerfile, switching back to root only as needed in the build process.

The Importance of Vulnerability Scanning

Vulnerability scanning is an important step in identifying vulnerabilities in a container image. Using tools such as Grype and Trivy, security and infrastructure engineers can routinely run security scans on open-source images in addition to images built via an automated pipeline. Creating an automated process to routinely scan the container image is a common part of most DevSecOps pipelines and will add greater visibility into vulnerability detection. Routine scanning also allows for the establishment of a vulnerability baseline and incremental mitigation.

Vulnerability scans typically yield a list of CVEs, which contain information about the vulnerability and any potential mitigations. A CVE usually has a Common Vulnerability Scoring System (CVSS) score, which assesses the severity of the vulnerability. Calculating or looking up the CVSS score can streamline the process of prioritizing the remediation of vulnerabilities.

Hardening of Docker images involves inspection of how the image is built, routinely scanning the built image for vulnerabilities, and then applying mitigations to the image building process. The continuous process of scanning new published images helps detect any new vulnerabilities. By establishing a routine process for hardening images, confidence is gained in the production pipeline starting from the development stage.