Whitelisting CVE’s

In last week’s blog, we covered how to create custom policies that can be used to evaluate your container images as part of your CI/CD pipeline or at any time during their lifetime. We explained that you should always perform a CVE scan of your container but that this is only the first step, in fact security vulnerabilities in the operating system packages are just the tip of the iceberg in terms of the tests that you should be performing.

Today we want to dig a little deeper into CVEs. Let’s start by looking at the NGINX image in the Anchore Navigator. NGINX is one of the most downloaded images on DockerHub and the chances are that if you are reading this blog you are running NGINX somewhere in your environment.

The following link will take you directly to the Security tab for the image to show you the CVEs found in this image.

At first glance, looking at the summary, you will probably be concerned: 18 high-level CVEs. The chances are that your policies are configured to fail the image and stop the deployment if any high-level CVEs are found but it’s not that simple.

Let’s drill down into the details of the first two entries in the list.

Here you can see that two packages include the following vulnerability: CVE-2016-7943 and that, currently, there is no fix available in the version of Debian that this container is built from. You may be tempted to ignore any CVE for which there is not yet a fix, however, there are may not be a fix because one is not required or perhaps there is a real vulnerability but the vendor has not yet released a patch. So let’s click on the link for CVE-2016-7943 to dig a little deeper.

This CVE has been issued against the libx11 library for X.org the graphical display server.
Libx11 is present in the image as it was pulled in by dependencies (libx11 -> libxpm4 -> nginx-module-image-filter).
The National Vulnerability Database (NVD), maintained by NIST, has this categorized as high severity. As you can see from the description this vulnerability may allow remote X servers to gain higher-level privileges. At the bottom of the page you will see the following notes:

Here you can see that the Debian security team has described it as a “minor issue”.
Given the Debian security team’s assessment, not to mention the fact that there will be no remote X window connections to this server we can safely ignore this high vulnerability.
That’s 1 down, 17 more to go …….

For the sake of this example, let’s presume that all the 18 high-level CVEs aren’t exploitable in our image. While that is certainly reassuring it’s also a lot of work for a DevOps engineer to review all CVEs in an image – there’s just too much noise and mistakes are bound to be made.

At Anchore we don’t believe that we should be producing lists of issues for engineers to review, the result of a policy evaluation should be a decision, does the image pass or fail? You will see that result on the Policy tab for this image.

We have evaluated this image with our default policy and with no whitelist, hence the failure.

The Anchore engine supports the notion of whitelists that allow users to define exceptions that should be ignored. For example, if we included CVE-2016-7943 in our whitelist then those first two high vulnerabilities would not have been shown in our policy evaluation.

Let’s move the command line to continue.

We’ll start by pulling down the latest nginx image and performing an analysis using Anchore.

# docker pull nginx:latest
# anchore analyze --image=nginx:latest

Next, we’ll create a very simple policy file called mypolicy, containing only two checks – for High and Critical CVEs

ANCHORESEC:VULNCRITICAL:STOP
ANCHORESEC:VULNHIGH:STOP

And then run a gate check against the image

# anchore gate --image=nginx:latest --policy=mypolicy
+--------------+------------------------+------------+----------+-----------------------------+-------------+
| Image Id     | Repo Tag               | Gate       | Trigger  | Check Output                | Gate Action |
+--------------+------------------------+------------+----------+-----------------------------+-------------+
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - libx11-data       |             |
|              |                        |            |          | (CVE-2016-7943 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2016-7943)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - libx11-6          |             |
|              |                        |            |          | (CVE-2016-7943 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2016-7943)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - libc-bin          |             |
|              |                        |            |          | (CVE-2014-9761 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2014-9761)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - multiarch-support |             |
|              |                        |            |          | (CVE-2014-9761 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2014-9761)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - libc6             |             |
|              |                        |            |          | (CVE-2014-9761 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2014-9761)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - libx11-data       |             |
|              |                        |            |          | (CVE-2016-7942 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2016-7942)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - libx11-6          |             |
|              |                        |            |          | (CVE-2016-7942 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2016-7942)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - libxml2           |             |
|              |                        |            |          | (CVE-2016-4448 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2016-4448)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - libvpx1           |             |
|              |                        |            |          | (CVE-2015-1258 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2015-1258)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - libtiff5          |             |
|              |                        |            |          | (CVE-2015-7554 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2015-7554)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - libtiff5          |             |
|              |                        |            |          | (CVE-2016-9535 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2016-9535)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - libxml2           |             |
|              |                        |            |          | (CVE-2016-1761 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2016-1761)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability found in | STOP        |
|              |                        |            |          | package - libtiff5          |             |
|              |                        |            |          | (CVE-2017-5225 - https      |             |
|              |                        |            |          | ://security-tracker.debian. |             |
|              |                        |            |          | org/tracker/CVE-2017-5225)  |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | FINAL      | FINAL    |                             | STOP        |
+--------------+------------------------+------------+----------+-----------------------------+-------------+

For the sake of this example let’s presume that all of these CVEs have been analyzed and based on the results whitelisted. We’ll create a file called mywhitelist that contains a line for each unique CVE along with the name of the gate: ANCHORESEC

Eg.

ANCHORESEC CVE-2014-9761
ANCHORESEC CVE-2015-1258
ANCHORESEC CVE-2015-7554
ANCHORESEC CVE-2016-1761
ANCHORESEC CVE-2016-2779
ANCHORESEC CVE-2016-3881
ANCHORESEC CVE-2016-4448
ANCHORESEC CVE-2016-6711
ANCHORESEC CVE-2016-6712
ANCHORESEC CVE-2016-7942
ANCHORESEC CVE-2016-7943
ANCHORESEC CVE-2016-9535
ANCHORESEC CVE-2017-0393
ANCHORESEC CVE-2017-5225

Now if we run the gate analysis passing the whitelist we’ll see a very different result.

# anchore gate --image=nginx:latest --policy=mypolicy --global-whitelist=mywhitelist
+--------------+------------------------+-------+---------+--------------+-------------+
| Image Id     | Repo Tag               | Gate  | Trigger | Check Output | Gate Action |
+--------------+------------------------+-------+---------+--------------+-------------+
| 5e69fe4b3c31 | docker.io/nginx:latest | FINAL | FINAL   |              | GO          |
+--------------+------------------------+-------+---------+--------------+-------------+

The gate command supports a –show-whitelisted flag that shows which allows a user to see which items were whitelisted and from which whitelist.

# anchore gate --image=nginx:latest --policy=mypolicy --global-whitelist=mywhitelist --show-whitelisted
+--------------+------------------------+------------+----------+-------------------------+-------------+-------------+
| Image Id     | Repo Tag               | Gate       | Trigger  | Check Output            | Gate Action | Whitelisted |
+--------------+------------------------+------------+----------+-------------------------+-------------+-------------+
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libx11-data             |             |             |
|              |                        |            |          | (CVE-2016-7943 - https  |             |             |
|              |                        |            |          | ://security-tracker.deb |             |             |
|              |                        |            |          | ian.org/tracker/CVE-201 |             |             |
|              |                        |            |          | 6-7943)                 |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libx11-6 (CVE-2016-7943 |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2016-7943)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libvpx1 (CVE-2017-0393  |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2017-0393)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libc-bin (CVE-2014-9761 |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2014-9761)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | multiarch-support       |             |             |
|              |                        |            |          | (CVE-2014-9761 - https  |             |             |
|              |                        |            |          | ://security-tracker.deb |             |             |
|              |                        |            |          | ian.org/tracker/CVE-201 |             |             |
|              |                        |            |          | 4-9761)                 |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libc6 (CVE-2014-9761 -  |             |             |
|              |                        |            |          | https://security-tracke |             |             |
|              |                        |            |          | r.debian.org/tracker/CV |             |             |
|              |                        |            |          | E-2014-9761)            |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libx11-data             |             |             |
|              |                        |            |          | (CVE-2016-7942 - https  |             |             |
|              |                        |            |          | ://security-tracker.deb |             |             |
|              |                        |            |          | ian.org/tracker/CVE-201 |             |             |
|              |                        |            |          | 6-7942)                 |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libx11-6 (CVE-2016-7942 |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2016-7942)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libvpx1 (CVE-2016-6711  |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2016-6711)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libvpx1 (CVE-2016-6712  |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2016-6712)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libxml2 (CVE-2016-4448  |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2016-4448)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libvpx1 (CVE-2015-1258  |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2015-1258)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libvpx1 (CVE-2016-3881  |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2016-3881)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libtiff5 (CVE-2015-7554 |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2015-7554)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libtiff5 (CVE-2016-9535 |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2016-9535)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | util-linux              |             |             |
|              |                        |            |          | (CVE-2016-2779 - https  |             |             |
|              |                        |            |          | ://security-tracker.deb |             |             |
|              |                        |            |          | ian.org/tracker/CVE-201 |             |             |
|              |                        |            |          | 6-2779)                 |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libxml2 (CVE-2016-1761  |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2016-1761)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | ANCHORESEC | VULNHIGH | High Vulnerability      | STOP        | global      |
|              |                        |            |          | found in package -      |             |             |
|              |                        |            |          | libtiff5 (CVE-2017-5225 |             |             |
|              |                        |            |          | - https://security-trac |             |             |
|              |                        |            |          | ker.debian.org/tracker/ |             |             |
|              |                        |            |          | CVE-2017-5225)          |             |             |
| 5e69fe4b3c31 | docker.io/nginx:latest | FINAL      | FINAL    |                         | GO          | none        |
+--------------+------------------------+------------+----------+-------------------------+-------------+-------------+

Whitelisting with Jenkins Plugin

The latest version of the Anchore plugin for Jenkins (version 1.0.9) includes the ability to pass a whitelist file to Anchore to the custom policy file. Using this mechanism you can include a whitelist in the workspace of your project that will be automatically picked up at analysis time.

To update to the latest version of Anchore login to the Jenkins web interface and select:

Manage Jenkins -> Manage Plugins

Press the “Check now button” to ensure that you have the latest plugin metadata.

From the “Updates” tab ensure that you’re upgrading to the latest Anchore plugin – at least version 1.0.9.

To use whitelists within your Jenkins project go to the Anchore Container Image Scanner build step where you will see a similar screen as shown above including the new “Global White list file” entry field.

From this screen press the “Save” button the new setting will not be honored until Save is pressed once.

The upcoming Anchore 2.0 release will support the graphical creation of whitelists and policies along with the ability to define a mapping file that allows the user to define which policies and whitelists are used for any given image based on its registry, repo name and tag.

In our next blog, we’ll dig deeper into advance policy and whitelist options as well as discussing curated whitelists.

Becoming a Container Security Champion

Since we released Anchore’s open source project almost a year ago we’ve seen fast-growing adoption by users who want to perform detailed inspection and analysis of their container images. By far the most common use case we see with our users is deploying Anchore within their continuous integration and deployment pipelines (CI/CD) especially with Jenkins.

In some of the recent events we’ve attended it’s been great to talk to end-users who are already using Anchore. We’ve heard a pretty consistent message in the conversations we’ve had:

Developers love Docker and it’s already a vital part of their development process and they are either already deploying Docker in production or are planning to do so. What we hear from operations and security folks is often a little different! We hear talks of ‘Shadow IT’ and unmanaged deployments. Right now it seems like the security and operations teams are racing to catch up with development.

Rather than trying to slow things down, most of the operations teams that we talk to want to just “get out of the way and let developers innovate” but they need to balance this with their organization’s needs around security and compliance.

Based on the experience we have already built with organizations in addressing these issues, today we are launching a new offering that we call Anchore Champion that provides a combination of services and support to jumpstart the process of securing an organization’s CI/CD pipeline and adding compliance and governance into their DevOps environment.

The Anchore Champion service begins with a container policy and compliance working session where we, virtually, get together with all the stakeholders: developers, operations and security to work through their requirements for compliance and then build a set of sample policies and whitelists that encompass their needs.

Next, we provide support for architecting a secure container build environment – helping integrate Anchore into a Jenkins or other CI/CD pipeline.

And we provide ongoing support for creating policies, configuration and general operation of Anchore.

Creating Policies

At the heart of Anchore’s solution is the concept of users certifying container images based on rules that they define. In the past certifications for applications typically came from operating systems vendors who defined their own standards and worked with independent software vendors (ISVs) on certification programs to give a level of assurance to end users that the application was compatible with the underlying operating system. Other organizations have created standards and certification tests to cover various forms of compliance validation, especially in the government sector or regulated industries.

Today the problem is more complex and there can no longer be just a single certification. For example, the requirements of a financial services company are different from the requirements of a healthcare company handling medical records and these are different from the needs of a federal institution and so on.

Anchore believes that rather than having certification in the hands of a small number of vendors or standards bodies, we want to allow organizations to define what certification means to them. In effect, we want to democratize certification.

Today we are seeing the baseline feature set for container security is a CVE scan and that’s certainly required but it’s just the first step.I’m sure the policies and you have in place today for your traditional deployments are more than just ensuring that you’ve updated all operating system packages.

These policies could cover security, starting with the ubiquitous CVE scan but then going further to analyze the configuration of key security components, for example, you could have the latest version of the apache webserver but have configured the wrong set of TLS Ciphers suites leading to insecure communication. Outside of security, policies could cover application-specific configurations to comply with best practices or to enable consistency and predictability.

In this blog, we will walk through some sample policies and cover how users can customize these policies as well as create and share their own policies.

Let’s start by looking at the policy evaluation of a test image. We will be using the anchore gate command. In Anchore’s terminology gates are checks that are run on images as they pass through the CI/CD pipeline or later when performing an evaluation on existing images.

Instead of using the default policy, we will use a customized policy called “basic-policy” that will be loaded by the CLI.

# anchore gate --image=testimage --policy=basic-policy
+--------------+------------------+-----------------+---------------+-------------------------+-------------+
| Image Id     | Repo Tag         | Gate            | Trigger       | Check Output            | Gate Action |
+--------------+------------------+-----------------+---------------+-------------------------+-------------+
| 9ebc746ba558 | testimage:latest | DOCKERFILECHECK | NOHEALTHCHECK | Dockerfile does not     | WARN        |
|              |                  |                 |               | contain any HEALTHCHECK |             |
|              |                  |                 |               | instructions            |             |
| 9ebc746ba558 | testimage:latest | PKGBLACKLIST    | PKGNAMEMATCH  | Package is blacklisted: | STOP        |
|              |                  |                 |               | openssh-server          |             |
| 9ebc746ba558 | testimage:latest | ANCHORESEC      | VULNMEDIUM    | Medium Vulnerability    | WARN        |
|              |                  |                 |               | found in package -      |             |
|              |                  |                 |               | bind-license            |             |
|              |                  |                 |               | (RHSA-2017:0276 - https |             |
|              |                  |                 |               | ://rhn.redhat.com/errat |             |
|              |                  |                 |               | a/RHSA-2017-0276.html)  |             |
| 9ebc746ba558 | testimage:latest | ANCHORESEC      | VULNMEDIUM    | Medium Vulnerability    | WARN        |
|              |                  |                 |               | found in package -      |             |
|              |                  |                 |               | openssl-libs            |             |
|              |                  |                 |               | (RHSA-2017:0286 - https |             |
|              |                  |                 |               | ://rhn.redhat.com/errat |             |
|              |                  |                 |               | a/RHSA-2017-0286.html)  |             |
| 9ebc746ba558 | testimage:latest | ANCHORESEC      | VULNMEDIUM    | Medium Vulnerability    | WARN        |
|              |                  |                 |               | found in package - vim- |             |
|              |                  |                 |               | minimal (RHSA-2016:2972 |             |
|              |                  |                 |               | - https://rhn.redhat.co |             |
|              |                  |                 |               | m/errata/RHSA-2016-2972 |             |
|              |                  |                 |               | .html)                  |             |
| 9ebc746ba558 | testimage:latest | ANCHORESEC      | VULNHIGH      | High Vulnerability      | STOP        |
|              |                  |                 |               | found in package -      |             |
|              |                  |                 |               | bind-license            |             |
|              |                  |                 |               | (RHSA-2017:0062 - https |             |
|              |                  |                 |               | ://rhn.redhat.com/errat |             |
|              |                  |                 |               | a/RHSA-2017-0062.html)  |             |
| 9ebc746ba558 | testimage:latest | IMAGECHECK      | BASEOUTOFDATE | Image base image        | WARN        |
|              |                  |                 |               | (docker.io/acathrow     |             |
|              |                  |                 |               | /aic-test:1a) ID is     |             |
|              |                  |                 |               | (9ebc746ba558), but the |             |
|              |                  |                 |               | latest ID for           |             |
|              |                  |                 |               | (docker.io/acathrow     |             |
|              |                  |                 |               | /aic-test:1a) is        |             |
|              |                  |                 |               | (f3e982542816)          |             |
| 9ebc746ba558 | testimage:latest | FINAL           | FINAL         |                         | STOP        |
+--------------+------------------+-----------------+---------------+-------------------------+-------------+

Here the output is formatted in a tabular view for a command line user to read however if you want to automate the processing of the output then the anchore command supports a --json or --plain command line option to output the results in a format that is easily parsed by other tools.

The most important part of the output is the last line that indicates that the final policy evaluation is “STOP”. Anchore gates will output one of three actions:

GO: The gate is open and the image should be allowed to pass through to the next stage.
STOP: The gate is closed and the image should not proceed
WARN: The gate is open and should proceed to the next step however warnings have been raised that should be reviewed.

If you are automating the use of Anchore from the command line then the return code from the anchore command can be used to evaluate the status: 0 = Go, 1 = Stop, 2 = Warn.

Looking at the output of the policy evaluation we can see that two policy checks outputted a “STOP” action. The first was due to a blacklisted package being present in the image and the second was due to a high-level CVE vulnerability.

Next, we’ll take a look at the policy.

DOCKERFILECHECK:NOTAG:STOP
DOCKERFILECHECK:NOFROM:STOP
DOCKERFILECHECK:NOHEALTHCHECK:WARN
DOCKERFILECHECK:EXPOSE:STOP:DENIEDPORTS=22
SUIDDIFF:SUIDFILEDEL:GO
SUIDDIFF:SUIDMODEDIFF:STOP
SUIDDIFF:SUIDFILEADD:STOP
IMAGECHECK:BASEOUTOFDATE:WARN
PKGBLACKLIST:PKGNAMEMATCH:STOP:BLACKLIST_NAMEMATCH=openssh-server
ANCHORESEC:FEEDOUTOFDATE:STOP:MAXAGE=2
ANCHORESEC:UNSUPPORTEDDISTRO:STOP
ANCHORESEC:VULNCRITICAL:STOP
ANCHORESEC:VULNHIGH:STOP
ANCHORESEC:VULNMEDIUM:WARN

The policy file lines are of the following format:

Gate name : Trigger : Action : Optional Parameters

You can consider a Gate as a family of checks that can be performed.

Checks can raise triggers that may have parameters.

Once a trigger is raised then an action (GO, STOP or WARN) is defined.

For example, looking at the following snippet from this simple policy:

DOCKERFILECHECK:NOHEALTHCHECK:WARN

Within the Dockerfilecheck gate there is a that looks for a healthcheck statement in the Dockerfile. If no healthcheck statement is found then the NOHEALTHCHECK trigger is raised.

In this example, we have configured Anchore to raise a warning if the Dockerfile does not include a health check.
In the next example we’ll blacklist two packages:

PKGBLACKLIST:PKGNAMEMATCH:STOP:BLACKLIST_NAMEMATCH=openssh-server,foolib

Here we have configured the PKGNAMEMATCH trigger to issue a STOP action if either openssh-server or foolib is present in the image.

The PKGBLACKLIST gate has two triggers : PKGFULLMATCH which matches both a package name and version and PKGNAMEMATCH which matches just the name of the package.

Anchore comes pre-installed with a number of policy modules that can be extended by the user and we regularly add new modules.

You can retrieve a full list of available policy options by running the following command:

# anchore gate --show-policytemplate

This will output a sample policy including all available gates including all available parameters.

The policy template option is useful for providing a policy file that you can further customize.

To get more detailed descriptions of gates, triggers and configuration options you can run the
following command:

# anchore gate --show-gatehelp
PKGCHECK:
   PKGNOTPRESENT:
     description: 'triggers if the package(s) specified in the params are not installed
       in the container image.  PKGFULLMATCH param can specify an exact match (ex:
       "curl|7.29.0-35.el7.centos").  PKGNAMEMATCH param can specify just the package
       name (ex: "curl").  PKGVERSMATCH can specify a minimum version and will trigger
       if installed version is less than the specified minimum version (ex: zlib|0.2.8-r2)'
     params: PKGFULLMATCH,PKGNAMEMATCH,PKGVERSMATCH

In this snippet, you can see the configuration options for the Package Check gate which allows a policy to specify that certain packages should be installed in the image. In addition to checking for the presence of a package, the user can configure minimum required versions.

We have set up a git repository to make it easier to share sample policies and allow the community to collaborate. You can access the repository here.

In our next blog, we’ll cover whitelists and show how you can reduce some of the noise that is often seen in CVE scans by whitelisting vulnerabilities that are not exploitable in your image.

Microservices -vs- MicroVMs

At Anchore we spend a whole lot of time looking at container images to provide detailed analysis and certification. Most of the discussions we hear in the industry around image analysis focus on CVE scanning: how many CVEs are in an image, what severity, etc. As we’ve mentioned before, we see CVE scanning as just the tip of the iceberg and that it’s possible to have all the latest operating system packages but still have an image that has security vulnerabilities or is otherwise not compliant with your operational, security or business policies.

There is another common issue in the tens of thousands of images that we’ve analyzed which we feel is more fundamental. As an industry we are moving to an architecture based on microservices and containers are really the key to enabling this. While the containers we’ve seen are often designed to run microservices, I’d argue that the majority of containers we see (both on DockerHub as well as our customer’s private images) are more like MicroVMs than Microservices. These images typically have a hundred or more packages and several thousand files.

In most cases, the images are general-purpose operating system images and differ only from their virtual machine brethren by not having a kernel installed. There has been much debate in the industry about image size and how smaller is better, allowing images to be rapidly deployed over the network. Others argue that size doesn’t matter and that the layered nature of Docker’s image format and caching largely mitigates this issue, but looking just at the size of the image doesn’t give you the complete picture.

While it’s certainly an important point to consider the real concern should be not the size of the image, but the content of the images.

Let’s take Alpine as an example. Let’s use the Anchore Navigator to view the contents and select the files tab to drill down further. Filtering this list to show the files in /bin highlights just how many executables are in the image. In a microservice, why does my image need utilities for process or file management? These come from having the busybox package in the image. While that may certainly be useful in some use cases I’d argue that having these kinds of binaries in an image that never directly calls them is an accident waiting to happen. I don’t mean to pick on Alpine which weighs in at 4MB (twenty-five times smaller than most base operating system images and certainly has less attack surface) but the point you must consider is that you must ensure that every artifact in your image serves a purpose and goes through some form of quality control to ensure that the final image is secure and meets your operational best practices.

Last month Oracle released a slimmed down Oracle Linux image which reduced the footprint down from 225MB to 114MB, you can read our analysis here, this week Red Hat upped the ante when they announced a slimmed down Red Hat Enterprise Linux Atomic Base Image.

The new RHEL image weighs in at 75MB, compared to 192MB for the standard RHEL image. In this image, Red Hat has removed a number of packages that are deemed not necessary for container deployments, two of the most interesting removals are systemd and Python. Traditionally all RHEL installs have included python since the YUM package manager is written in Python. To get around this Red Hat has created a new mini package manager called microdnf. While microdnf is not as functional as YUM or as DNF, the next generation package manager for RHEL based distributions, it does just what is needed: install, remove and update packages.

I wanted to look at what else changed in the image so I pulled the RHEL Atomic image from Red Hat’s registry. If you don’t have access to the RHEL registry, you can take a look at the analysis of the image using the Anchore navigator here:

Note: This image is not available publicly on DockerHub.

For the rest of the analysis, I’m going to use Anchore’s command line tools.

First I need to analyze the image.

# anchore analyze --image=registry.access.redhat.com/rhel7-atomic

I’ve already analyzed the standard rhel 7 image so now I want to run a query to compare the packages installed in the RHEL Atomic image with the standard RHEL image using the show-pkg-diffs query.

# anchore query --image=registry.access.redhat.com/rhel7 show-pkg-diffs registry.access.redhat.com/rhel7-atomic

 

Package RHEL 7 RHEL Atomic
python-chardet 2.2.1-1.el7_1 Not Installed
librhsm Not Installed 0.0.1-1.el7
yum-plugin-ovl 1.1.31-40.el7 Not Installed
libuser 0.60-7.el7_1 Not Installed
json-glib Not Installed 1.0.2-1.el7
python-urlgrabber 3.10-8.el7 Not Installed
libblkid 2.23.2-33.el7 Not Installed
audit-libs 2.6.5-3.el7_3.1 Not Installed
libsolv Not Installed 0.6.20-5.el7
xz 5.2.2-1.el7 Not Installed
file-libs 5.11-33.el7 Not Installed
rpm-build-libs 4.11.3-21.el7 Not Installed
python-libs 2.7.5-48.el7 Not Installed
qrencode-libs 3.4.1-3.el7 Not Installed
gdbm 1.10-8.el7 Not Installed
cryptsetup-libs 1.7.2-1.el7 Not Installed
dbus-libs 1.6.12-17.el7 Not Installed
tar 1.26-31.el7 Not Installed
dbus-glib 0.100-7.el7 Not Installed
cracklib-dicts 2.9.0-11.el7 Not Installed
kmod 20-9.el7 Not Installed
systemd 219-30.el7_3.7 Not Installed
subscription-manager 1.17.15-1.el7 Not Installed
libpwquality 1.2.3-4.el7 Not Installed
pygpgme 0.3-9.el7 Not Installed
python-dmidecode 3.10.13-11.el7 Not Installed
pyliblzma 0.5.3-11.el7 Not Installed
device-mapper 1.02.135-1.el7_3.3 Not Installed
kmod-libs 20-9.el7 Not Installed
shadow-utils 4.1.5.1-24.el7 Not Installed
python-pycurl 7.19.0-19.el7 Not Installed
libcap-ng 0.7.5-4.el7 Not Installed
python-rhsm-certificates 1.17.9-1.el7 Not Installed
kpartx 0.4.9-99.el7_3.1 Not Installed
python-iniparse 0.4-9.el7 Not Installed
microdnf Not Installed 2-3.el7.1.1
pam 1.1.8-18.el7 Not Installed
cracklib 2.9.0-11.el7 Not Installed
procps-ng 3.3.10-10.el7 Not Installed
pyxattr 0.5.1-5.el7 Not Installed
vim-minimal 7.4.160-1.el7_3.1 Not Installed
python 2.7.5-48.el7 Not Installed
python-rhsm 1.17.9-1.el7 Not Installed
python-ethtool 0.8-5.el7 Not Installed
cpio 2.11-24.el7 Not Installed
libutempter 1.1.6-4.el7 Not Installed
device-mapper-libs 1.02.135-1.el7_3.3 Not Installed
systemd-libs 219-30.el7_3.7 Not Installed
dmidecode 3.0-2.el7 Not Installed
m2crypto 0.21.1-17.el7 Not Installed
hardlink 1.0-19.el7 Not Installed
rpm-python 4.11.3-21.el7 Not Installed
yum-utils 1.1.31-40.el7 Not Installed
dbus-python 1.1.1-9.el7 Not Installed
python-dateutil 1.5-7.el7 Not Installed
librepo Not Installed 1.7.16-1.el7
util-linux 2.23.2-33.el7 Not Installed
usermode 1.111-5.el7 Not Installed
yum-metadata-parser 1.1.4-10.el7 Not Installed
pygobject3-base 3.14.0-3.el7 Not Installed
dracut 033-463.el7 Not Installed
rootfiles 8.1-11.el7 Not Installed
ustr 1.0.4-16.el7 Not Installed
elfutils-libs 0.166-2.el7 Not Installed
diffutils 3.3-4.el7 Not Installed
dbus 1.6.12-17.el7 Not Installed
libuuid 2.23.2-33.el7 Not Installed
gdb-gdbserver 7.6.1-94.el7 Not Installed
libmount 2.23.2-33.el7 Not Installed
libxml2-python 2.9.1-6.el7_2.3 Not Installed
yum 3.4.3-150.el7 Not Installed
virt-what 1.13-8.el7 Not Installed
libdnf Not Installed 0.7.4-2.el7.el
libsemanage 2.5-5.1.el7_3 Not Installed
gzip 1.5-8.el7 Not Installed
passwd 0.79-4.el7 Not Installed
python-kitchen 1.1.1-5.el7 Not Installed
libnl 1.1.4-3.el7 Not Installed
binutils 2.25.1-22.base.el7 Not Installed
acl 2.2.51-12.el7 Not Installed

Here you’ll see there are 80 package differences. Six packages have been added to support the new package manager: librhsm, json-glib, libsolv, microdnf, librepo and libdnf.74 packages have been removed leaving just the minimum set of packages.

Out of interest, I wanted to see how this package list differed from Oracle’s slim image.

Package Oracle Linux Slim RHEL Atomic
python-chardet 2.2.1-1.el7_1 Not Installed
nss-tools 3.21.3-2.0.1.el7_3 3.21.3-2.el7_3
python-urlgrabber 3.10-8.el7 Not Installed
libxml2 2.9.1-6.0.1.el7_2.3 2.9.1-6.el7_2.3
audit-libs 2.6.5-3.el7 Not Installed
nss-sysinit 3.21.3-2.0.1.el7_3 3.21.3-2.el7_3
file-libs 5.11-33.el7 Not Installed
rpm-build-libs 4.11.3-21.el7 Not Installed
python-libs 2.7.5-48.0.1.el7 Not Installed
json-glib Not Installed 1.0.2-1.el7
gdbm 1.10-8.el7 Not Installed
nss 3.21.3-2.0.1.el7_3 3.21.3-2.el7_3
pyxattr 0.5.1-5.el7 Not Installed
yum-plugin-ovl 1.1.31-40.el7 Not Installed
basesystem 10.0-7.0.1.el7 10.0-7.el7
pygpgme 0.3-9.el7 Not Installed
coreutils 8.22-18.0.1.el7 8.22-18.el7
shadow-utils 4.1.5.1-24.el7 Not Installed
python-pycurl 7.19.0-19.el7 Not Installed
libcap-ng 0.7.5-4.el7 Not Installed
bash 4.2.46-21.0.1.el7_3 4.2.46-21.el7_3
python-iniparse 0.4-9.el7 Not Installed
microdnf Not Installed 2-3.el7.1.1
librhsm Not Installed 0.0.1-1.el7
kernel-container 3.10.0-0.0.0.2.el7 Not Installed
gobject-introspection Not Installed 1.42.0-1.el7
python 2.7.5-48.0.1.el7 Not Installed
cpio 2.11-24.el7 Not Installed
yum-utils 1.1.31-40.el7 Not Installed
pyliblzma 0.5.3-11.el7 Not Installed
rpm-python 4.11.3-21.el7 Not Installed
librepo Not Installed 1.7.16-1.el7
yum-metadata-parser 1.1.4-10.el7 Not Installed
libsolv Not Installed 0.6.20-5.el7
ustr 1.0.4-16.el7 Not Installed
oraclelinux-release 7.3-1.0.4.el7 Not Installed
diffutils 3.3-4.el7 Not Installed
redhat-release-server 7.3-7.0.1.el7 7.3-7.el7
libxml2-python 2.9.1-6.0.1.el7_2.3 Not Installed
yum 3.4.3-150.0.1.el7 Not Installed
libdnf Not Installed 0.7.4-2.el7.el
libsemanage 2.5-5.1.el7_3 Not Installed
python-kitchen 1.1.1-5.el7 Not Installed
gpg-pubkey ec551f03-53619141 Not Installed

Ignoring the version differences, there are 7 packages in RHEL Atomic not present in the Oracle Slim image which support the new microdnf package manager. There are 28 packages in Oracle Slim that are not in the RHEL Atomic image – unsurprisingly most of these relate to the inclusion of YUM. It will be interesting to see if Oracle Linux and the other RHEL derivatives follow suit and use microdnf in their images.

This is a great step forward for RHEL users, reducing the image size and the attack surface but still leaves a lot of, arguably, unnecessary content in the image. Take a look at the files view in the content tab of the Anchore Navigator for this image here and, as we did for Alpine earlier, filter for /bin to see the utilities and other libraries installed in the image.

At this point, the challenge in reducing the image further is that most of the packages left are required either in whole or more likely in part due to dependencies. For example, you could argue that there is no good reason to have the /bin/chmod command in the image however that is part of the coreutils package which is required by multiple other packages so any further steps forward will require some major changes in packaging.

If you have a Red Hat Enterprise Linux subscription I’d encourage you to check out the new Atomic base image and see how you can reduce the footprint and attack surface of your RHEL based images.

And whether you use RHEL, CentOS, Debian, Ubuntu, Alpine or other distributions you can use Anchore’s image analysis and compliance tools to ensure that the images you deploy meet your security and best practices requirements.

Improved Jenkins Integration

Today we have released an update to our popular open source Jenkins plugin adding a number of powerful new features.

Using Anchore’s freely available and open source Jenkins plugin you can secure your Jenkins pipeline in less than 30 minutes adding image scanning including not just CVE based security scans but policy-based scans that can include checks around security, compliance and operational best practices.

The first new feature to highlight is an updated user interface that improves both the aesthetic and the functionality of the UI. In the first screenshot below you can see that while the build has succeeded we have raised a number of warnings.

  •     The container was built from a base image with the tag latest rather than from a specific named tag
  •     The Dockerfile does not include any health check instructions which would simplify ongoing monitoring of the service.
  •     The acme-logging package has not been installed which is a recommended package for this organization.

Policies are customizable, along with whitelists, and are typically defined by the Security or Operations team.

The Policy evaluation summary is always produced by the Anchore plugin, however, there are other reports that a user can define to be run during the CI/CD pipeline.

In the first example, you can see a package manifest that has been produced – both in the form of a searchable web interface but also as a JSON file in the Jenkins project workspace that contains machine-readable output.

In the final example, we see a report detailing the difference in packages between the base image and the final image produced by the build.


Select:   Manage Jenkins > Manage Plugins > Updates

If you are already running the Anchore Jenkins plugin then you can update the Anchore plugin directly from the Jenkins web interface. At the time of writing the latest version of the plugin is version 1.0.7.

If you are not running Anchore’s plugin, there is detailed instructions on the following page.

The second interesting new feature is support for Jenkins Pipelines. In our previous examples, we have illustrated the use of Anchore within a Jenkins Freestyle Project which is the traditional way of architecting a Jenkins build, using the Jenkins web interface to define projects, adding build steps, scripts, etc.

In the Pipeline model, the entire build process is defined as code in a Jenkinsfile. This file can be created, edited and managed in the same way as any other artifact of your software project. For example, you can check your pipeline definition into your source control system, dynamically create the build instructions based on the configuration of your application or perform countless other forms of automation.

Pipeline builds can be more complex including forks/joins and parallelism. The pipeline is more resilient and can survive the master node failure and restarts. Pipelines are written in Groovy scripts and to add an Anchore scan you need to add the following simple code snippet.

node {
def imageLine = IMAGE + ' ' + env.WORKSPACE + '/DockerFile'
writeFile file: 'anchore_images', text: imageLine
anchore name: 'anchore_images', policyName: 'anchore_policy', bailOnFail: false, inputQueries: [[query: 'list-packages all'], [query: 'cve-scan all']]
}

Here the IMAGE is the ID of the container image that was just created. This could be in the form of an image ID (short or long-form), for example, 67591570dd29. Or the REPO/TAG can be used -for example, webapp/frontend:123456.

This code snippet writes out the anchore_images file that is used by the plugin to define which images are to be scanned.

The Dockerfile is read from the project workspace as is the file containing the policy that you wish to evaluate against the image, in this case, we have called the policy file anchore_policy and have stored this file in the project’s workspace.

This code snippet can be crafted by hand or built using the Jenkins UI.

Select:  Pipeline Syntax from the Project

This will launch the Snippet Generator where you can enter the required parameters and press the Generate Pipeline Script button which will produce the required snippet.

It’s quick and easy to add Image scanning and policy to your Jenkins project and we’re here to help.

If you have any questions or would like to learn more you can join our slack channel by clicking the button below or fill out the form to send us a direct message.