For most users analyzing or auditing container images usually means running a CVE scan and while that is certainly required, it should be just the first step. Anchore supports creating policies that can be used to assess the compliance of your containers, these policy checks could cover security, starting with the ubiquitous CVE scan but then going further to analyze the configuration of key security components, for example, you could have the latest version of the apache webserver but have configured the wrong set of TLS Ciphers suites leading to insecure communication. Outside of security, policies could cover application-specific configurations to comply with best practices or to enable consistency and predictability.
Today there are many tools that can perform CVE scans of a container image however when we speak to users we often hear that either they do not perform these scans or if they do they do not gate container deployments based on the results of these scans. When we asked these users why they didn’t stop their deployment based on the CVE scanner’s results, we were told “if we did then we’d not deploy any containers – they all fail! “
This is a common issue, for example, if you look at the official images on Docker Hub or Docker Store for CentOS, Debian, Oracle or Ubuntu they all appear to have high or critical vulnerabilities many of which are unfixed, some appear to be CVEs that are unresolved for over a year or more.
We have covered this topic previously with respect to CentOS where we saw many vulnerabilities reported by other tools in the CentOS image that were not accurate and similar issues with Oracle and RHEL images.
It has been pointed out that Debian, which as we discussed in our previous blog, is the most popular operating system used on Docker Hub seems to have the most vulnerabilities. Regardless of the CVE scanner used, the Debian image looks insecure with many unpatched vulnerabilities, but looks can be deceptive.
We will take a look at the Debian image and discuss the results found by various scanners, explain the differences in results and show how you can remove the noise and get a clear view of the security of your containers.
Which Package is Vulnerable?
Let’s start by looking at a vulnerability reported in the latest Debian image: CVE-2017-12424 which describes a vulnerability in the shadow project which provides tools and libraries for maintaining the password database.
Looking at the output of most of the CVE scanners you will see output similar to the following:
Here we can see that shadow version 4.4-4.1 is installed and vulnerable to the critical severity CVE 2017-12424. But if you look for the shadow package in your image you will not find it.
root@debian:/# dpkg -s shadow dpkg-query: package 'shadow' is not installed and no information is available
So if you try to upgrade that individual package you’ll receive an error.
Debian reports CVEs against source packages rather than against binary packages so in this example while the source package was shadow the binary package is called passwd.
You can look up the source package for a given binary package either using the dpkg utility or using apt-get source.
root@debian:/# dpkg -s passwd | grep Source Source: shadow
This example using the shadow package is rather straight forward, I would expect that most readers of this article would quickly work out the mapping to the passwd binary package however in many other cases things are not so simple. For example, many non-kernel binaries are created from the Linux source package this leads to some tools reporting kernel CVEs in a container image that includes no kernel.
For this reason in the Anchore Cloud and Anchore Engine both report on the binary package not the underlying source package.
How Many Vulnerabilities?
Analyzing the same image with different scanners often results in very different numbers of reported vulnerabilities. In some cases, as we described in a previous blog, this may be a result of the scanner not taking into backporting of fixes, or not using distribution’s own security vulnerability feed. In other cases the mapping of CVE to source and image package causes confusion. For example, looking at the current debian:latest image using Anchore’s scanner we can see eight packages are shown as being vulnerable to the vulnerability described in CVE 2016-2779.
This CVE was reported against the util-linux source package. The binary packages listed in this report are all built from the util-linux source package. A tool, such as Anchore, that reports on binary packages will report seven packages against that CVE while a tool that reports on source packages may only report one. Whether one or all of these binary packages are vulnerable to the vulnerability described in the CVE is something that requires digging deeper into the bug reports and mailing list traffic. Ideally, the distributions would provide more binary specific details in the vulnerability data to assist in this mapping.
A more interesting question is: if even one of these packages was vulnerable why after nearly 18 months are fixes not available for these packages? We need to dig deeper.
Is it Really Vulnerable?
As we saw in the previous example we often see unfixed CVEs in images. For example in the current debian:latest image we see 50 vulnerabilities that have no fixes and 12 of these are rated as High severity.
Even if we just counted the number of unique CVEs, not packages, we still see: three high, three medium, one low and 15 Negligible CVEs.
Why is that number so high? Since this is the latest official Debian image there is certainly more to this, especially given the fact that the Debian security team are renowned for their focus and responsiveness.
We can start by looking at the CVE in Debian’s security tracker: CVE-2016-2779. Here you can see that the current stable version of Debian, stretch, is classified as being vulnerable, however, looking in the notes section we see the following:
The security team notes that no Debian Security Advisory will be issued for this vulnerability (no-dsa). You can read more about this on the Debian Security FAQ here, however, the concept here is that while a source package may be vulnerable the way that it is compiled or deployed may mitigate the issue. In some cases, this may be because the package is built with specific compile-time options that don’t trigger the security issue in other cases this may be due to the environment in which it is run. In this specific case, it was decided that the best approach was to address the underlying issue in the Linux kernel so that no version of this package could trigger the vulnerability.
Based on this data we should not be concerned about this high criticality CVE in our scan results. There is certainly an argument to be made that given this fact maybe this CVE should not be reported in the Debian vulnerability feed, which is the approach that the Red Hat-based distributions such as CentOS, Oracle and RHEL take. I have not looked into the history around this decision but can imagine the strong arguments to be made on both sides. Presumably anticipating this issue, the Debian team includes metadata in their vulnerability feed that indicates the No Debian Security Advisory decision and commentary. This is data we can then use as part of our analysis of the image.
Looking at the current debian:latest image using Anchore Cloud we can view the image’s policy status to see if it PASSES or FAILES based on the default image policy.
Here we can see that the image has failed, scrolling down we can see 12 High criticality vulnerabilities that led to this result. You can read more about the image policies here.
Anchore includes support for whitelists which allows for certain policy checks, such as select CVEs, to be suppressed. A CVE may be present in a package but not exploitable in that package’s configuration as we saw earlier in this blog. So using the Anchore policy editor a user can create and manage whitelists to filter out false positives.
Whitelists can be created and managed in the policy editor or from the image’s policy view but in the case of Debian security advisories it is easier to create a whitelist and upload into the Anchore Cloud.
We have published a simple utility that creates whitelists based on the data published in the Debian security tracker. You can clone this utility from our public GitHub repository.
Running the debian-whitelist.py utility will create a JSON document for each of the current Debian releases: Wheezy, Jessie, Stretch and Buster (which right now has no whitelisted CVEs).
1. Create a free account on the Anchore Cloud
2. Open the Policy editor by selecting the menu icon on the left navigation menu.
3. Expand the Whitelist editor
4. For each whitelist press the “Upload Whitelist Item” button and upload the JSON document. The whitelists will be named based on the version of Debian and the date. These names can be edited to be more user friendly.
You will now have whitelists for each Debian version.
Next, we need to use the Mapping Editor to define what whitelist is used for a specific image.
5. Expand the Mapping Editor
6. Select “Create New Mapping” to create a new mapping.
7. Give your mapping a name, eg. “Debian latest”
8. Specify library/debian as the repository name
9. Specify latest as the tag
10. Select the whitelist you created from the dropdown.
11. Select “Save All” to save the whitelist and policy mapping.
Now when you view the Debian image you will see that a user-defined policy has been used and that the image passes.
The whitelists CVEs can be viewed by checking the Show Whitelisted entries checkbox.
Currently, Anchore only applies the whitelist to the policy view and not to the list of CVEs presented in the security tab which shows all CVEs present in the image.
By using the default policy we are just performing basic CVE policy checks on the image but using the policy editor you can create policies that do much more.
