2017 | Page 3 of 3

Slimming Down Images

Posted on February 22, 2017April 13, 2022 by Christian Wiens

Oracle just announced a new container image: Oracle Linux 7-Slim.

Their goal was to create a more lean image and improve security in the process, since reducing the footprint of the container also reduces the attack surface.

You can check out that image here using Anchore Navigator where you can see that the image weighs in at a little over 100MB, compared to the standard Oracle Linux image which is over twice that size. While that’s nowhere near as small as Alpine, which is a minuscule 4MB, Oracle’s base image is much smaller than the other major Linux distros.

The Anchore service, which powers the Navigator, tracks the most popular images on DockerHub along with images requested by registered users, so when a new image is published we pull down the image and perform our detailed analysis. From that data we can tell that Oracle does a good job of regularly updating their base image and usually this image has no security vulnerabilities (CVEs) as it’s updated frequently. You can subscribe to any image on the Navigator to receive notifications when the TAGs are updated – for example when Oracle updated their standard image on the 21st of February all users who subscribed to that image received email notification.

Last month we blogged about how you can use Anchore to compare images to see what has changed so today we took a look at the new Oracle slim image to see how Oracle shaved around 100MB off the image.

For those who want to follow along you can use the following command:

# anchore query --image=oraclelinux show-pkg-diffs oraclelinux:7-slim

Package	Oracle Linux	Oracle Linux Slim
procps-ng	3.3.10-10.el7	Not Installed
openssh-clients	6.6.1p1-33.el7_3	Not Installed
libuser	0.60-7.el7_1	Not Installed
oracle-logos	70.0.3-4.0.7.el7	Not Installed
tar	1.26-31.el7	Not Installed
json-c	0.11-4.el7_0	Not Installed
iputils	20160308-8.el7	Not Installed
pygobject2	2.28.6-11.el7	Not Installed
rhnsd	5.0.13-5.0.1.el7	Not Installed
rhn-check	2.0.2-8.0.4.el7	Not Installed
xz	5.2.2-1.el7	Not Installed
iproute	3.10.0-74.0.1.el7	Not Installed
libmnl	1.0.3-7.el7	Not Installed
python-hwdata	1.7.3-4.el7	Not Installed
rsyslog	7.4.7-16.0.1.el7	Not Installed
bind-license	9.9.4-38.el7_3.2	Not Installed
pam	1.1.8-18.el7	Not Installed
acl	2.2.51-12.el7	Not Installed
dbus-glib	0.100-7.el7	Not Installed
cracklib-dicts	2.9.0-11.el7	Not Installed
vim-minimal	7.4.160-1.el7_3.1	Not Installed
systemd	219-30.0.1.el7_3.6	Not Installed
libpwquality	1.2.3-4.el7	Not Installed
libnetfilter_conntrack	1.0.4-2.el7	Not Installed
python-dmidecode	3.10.13-11.el7	Not Installed
newt-python	0.52.15-4.el7	Not Installed
hostname	3.13-3.el7	Not Installed
libestr	0.1.9-2.el7	Not Installed
device-mapper	1.02.135-1.el7_3.2	Not Installed
rhnlib	2.5.65-2.0.1.el7	Not Installed
passwd	0.79-4.el7	Not Installed
yum-rhn-plugin	2.0.1-6.0.1.el7	Not Installed
kpartx	0.4.9-99.el7_3.1	Not Installed
libblkid	2.23.2-33.0.1.el7	Not Installed
dracut	033-463.0.1.el7	Not Installed
python-gudev	147.2-7.el7	Not Installed
policycoreutils	2.5-11.0.1.el7_3	Not Installed
cracklib	2.9.0-11.el7	Not Installed
iptables	1.4.21-17.el7	Not Installed
fipscheck	1.4.1-5.el7	Not Installed
yum-plugin-ulninfo	0.2-13.el7	Not Installed
dbus-libs	1.6.12-17.0.1.el7	Not Installed
kmod	20-9.el7	Not Installed
openssh-server	6.6.1p1-33.el7_3	Not Installed
GeoIP	1.5.0-11.el7	Not Installed
systemd-libs	219-30.0.1.el7_3.6	Not Installed
python-ethtool	0.8-5.el7	Not Installed
bind-libs-lite	9.9.4-38.el7_3.2	Not Installed
libutempter	1.1.6-4.el7	Not Installed
device-mapper-libs	1.02.135-1.el7_3.2	Not Installed
sysvinit-tools	2.88-14.dsf.el7	Not Installed
m2crypto	0.21.1-17.el7	Not Installed
hardlink	1.0-19.el7	Not Installed
libgudev1	219-30.0.1.el7_3.6	Not Installed
dbus-python	1.1.1-9.el7	Not Installed
dhcp-libs	4.2.5-47.0.1.el7	Not Installed
slang	2.2.4-11.el7	Not Installed
util-linux	2.23.2-33.0.1.el7	Not Installed
usermode	1.111-5.el7	Not Installed
libnl	1.1.4-3.el7	Not Installed
newt	0.52.15-4.el7	Not Installed
dhclient	4.2.5-47.0.1.el7	Not Installed
libnfnetlink	1.0.1-4.el7	Not Installed
qrencode-libs	3.4.1-3.el7	Not Installed
rootfiles	8.1-11.el7	Not Installed
elfutils-libs	0.166-2.el7	Not Installed
libedit	3.0-12.20121213cvs.el7	Not Installed
tcp_wrappers-libs	7.6-77.el7	Not Installed
pyOpenSSL	0.13.1-3.el7	Not Installed
openssh	6.6.1p1-33.el7_3	Not Installed
dbus	1.6.12-17.0.1.el7	Not Installed
libuuid	2.23.2-33.0.1.el7	Not Installed
logrotate	3.8.6-12.el7	Not Installed
dhcp-common	4.2.5-47.0.1.el7	Not Installed
cryptsetup-libs	1.7.2-1.el7	Not Installed
libmount	2.23.2-33.0.1.el7	Not Installed
initscripts	9.49.37-1.0.1.el7	Not Installed
kmod-libs	20-9.el7	Not Installed
rhn-client-tools	2.0.2-8.0.4.el7	Not Installed
hwdata	0.252-8.4.el7	Not Installed
gzip	1.5-8.el7	Not Installed
fipscheck-lib	1.4.1-5.el7	Not Installed
libselinux-utils	2.5-6.el7	Not Installed
binutils	2.25.1-22.base.el7	Not Installed
rhn-setup	2.0.2-8.0.4.el7	Not Installed

Here you can see that 85 packages were removed from the standard image. Some of the removals are obvious optimizations – removing unneeded utilities and libraries and others are notable as they highlight some interesting issues in the regular image – for example, openssh-server has been removed – which you might argue has no business being installed in a container image in the first place.

There are other changes such as the removal of dbus and kmod that really go to highlight how many containers are being built today. I’d argue that in many cases organizations aren’t deploying microservices they are deploying microVMs. Many images look like a whole operating system but just packaged up in a Docker image. There’s a lot of other fat that can be trimmed from most containers – for example take a look at the contents of this image: navigate to the contents tab, look at the files view and filter for /bin and while you scroll through the 51 pages ask if these binaries are really needed in your image.

There’s a lot of work still to be done by most Linux distro vendors to build more efficient and more secure images. Removing selected RPMs and DEBs helps but the size and scope of many of the operating system packages still lead to more content being installed that is required.

One cautionary note:

While size certainly does matter it should not be your only consideration in selecting a base image to use from DockerHub or any other registry.

Ensure that the image is well maintained – for example, check that it gets updated frequently enough to meet your needs. Is the content coming from known-good sources? You certainly don’t want to bring in packages from an unknown origin. Are the operating system packages being maintained and tested including security fixes with published CVE security feeds, is the default out-of-the-box configuration secure?

Anchore can help you answer those questions – whether it’s by using the Navigator to pre-screen images for security issues and to view update history or by building custom policies that define your own rules for certifying your containers.

Keeping Secrets

Posted on February 17, 2017December 20, 2021 by Christian Wiens

Docker recently announced an exciting new release of Docker Datacenter that included Integrated Secrets Management from Docker 1.13. Many containers need access to sensitive information as part of their configuration, for example, they may need the password to access a database or the API key to access web services. These secrets need to be securely passed to the running container. In the past various other mechanisms have been used to pass secrets including using environment variables and volume mounting files from the host into the container. Each of these, and other, alternatives have their own individual drawbacks but all share the same issue: they store unencrypted secrets on the host that an administrator may be able to see. There are other solutions that can be used to securely manage secrets, for example, the popular Vault project from HashiCorp, however, having integrated secrets management is a great step forward.

As many organizations are now moving away from legacy approaches such as environment variables and volumes to pass secrets to using Docker’s new integrated secrets management or 3rd party solutions such as Vault it is important to ensure that you are not already inadvertently including sensitive information such as passwords, certificates and API keys within your image.

During testing and development, it is very easy to leave artifacts such as private certificates or keys within your image to simplifying testing and in many cases these can inadvertently be carried forward into your production deployment. The most famous example of this occurred last summer when Twitter’s now-defunct Vine service was analyzed by a security researcher who found that they had mistakenly disabled authentication on their Internet-facing Docker registry. The researcher was able to pull Vine’s images down to his laptop and inspect them. Within these images, he found API keys, source code and other secrets.

While many users are scanning their images for CVEs, an image may pass this basic check but may still be insecure, misconfigured or in some other way out of compliance. Container images typically contain hundreds, often thousands of files – some coming from operating system packages, some from configuration files, some from 3rd party software libraries such as Node.JS NPMs, Ruby GEMs, Python modules, Java Archives, and some may be supplied by the user. Each one of these artifacts should undergo the same level scrutiny as the operating system packages. One of the critical checks that should be performed before an image is deployed is to ensure that it does not contain source code, sensitive configuration information and secrets such as API keys and passwords.

Anchore takes a very different approach to image security than traditional image scanners that look for CVEs. Using Anchore users can define policies that specify rules to govern security vulnerabilities, package whitelists and blacklists, configuration file contents, presence of credentials in an image, manifest changes, exposed ports or any user-defined checks. These policies can be deployed site-wide or customized for specific images or categories of applications.

You can read more about Anchore’s policy-based approach, however, if you want to take a more practical approach you can use open source Anchore Engine to inspect your own images and look for secrets. The following guide will walk you through setting up Anchore and analyzing your images, it should take no more than 10 minutes.

There are a number of ways to install Anchore including using operating system packages, PIP or even via a container. In this example, I’m using a CentOS 7 host that is already running Docker. If the system is not already configured to use the Extra Packages for Enterprise Linux (EPEL) repository then I need to run:

# yum install epel-release

Installing Anchore is as simple as installing a YUM repo file and then installing a single package.

# yum install http://repo.ancho.re/anchore/1.1/centos/7/noarch/anchore-release-1.1.0-1.el7.centos.noarch.rpm
# yum install anchore

At this point Anchore is installed, all we need to do now is run a sync to download the latest security data from the Anchore service.

# anchore feeds sync

Now we are ready to analyze containers.
Presuming the container image has been pulled to the localhost you can simply run the analyze command. In my example, I’m analyzing the myapp:latest image.

# anchore analyze --image=myapp:latest

If the Dockerfile is available then you can pass the Dockerfile to the anchore command, this provides a little more information to the analysis routine but is not required.

# anchore analyze --image=myapp:latest --dockerfile=/path/to/my/Dockerfile

We can now run a policy check on the image.
The default policy does not perform any checks for secrets but we can easily add that check.
Create a file named mypolicy and enter the following lines:

DOCKERFILECHECK:NOTAG:STOP
DOCKERFILECHECK:SUDO:GO
DOCKERFILECHECK:EXPOSE:STOP:DENIEDPORTS=22
DOCKERFILECHECK:FROMSCRATCH:WARN
DOCKERFILECHECK:NOFROM:STOP
SUIDDIFF:SUIDFILEDEL:GO
SUIDDIFF:SUIDMODEDIFF:STOP
SUIDDIFF:SUIDFILEADD:STOP
PKGDIFF:PKGVERSIONDIFF:STOP
PKGDIFF:PKGADD:WARN
PKGDIFF:PKGDEL:WARN
ANCHORESEC:VULNUNKNOWN:GO
ANCHORESEC:VULNHIGH:STOP
ANCHORESEC:VULNMEDIUM:WARN
ANCHORESEC:VULNLOW:GO
ANCHORESEC:VULNCRITICAL:STOP
ANCHORESEC:UNSUPPORTEDDISTRO:WARN
FILECHECK:FILENAMEMATCH:STOP:FILECHECK_NAMEREGEXP=.*/.ssh/id_rsa$

The last line uses the FILECHECK policy check. Here we are looking at a list of all the files in the image and using a regular expression to look for any private ssh keys in the image.
The FILECHECK policy module (or gate) can do matching on filenames or on filecontents, for example looking for specific strings within any file in the image. In this example, we are simply looking for ssh keys.

# anchore gate  --image=myapp:latest --policy=/path/to/mypolicy

In my test image policy check returns two lines. The last line gives the final result of the policy check issuing a “STOP” meaning that the image has failed. The first line shows the policy that triggered this failure. Depending on your image you may see more or fewer policy violations.

+--------------+-------------------+-----------+---------------+-------------------+-------------+
| Image Id     | Repo Tag          | Gate      | Trigger       | Check Output      | Gate Action |
+--------------+-------------------+-----------+---------------+-------------------+-------------+
| 9f767c5486f4 | aic-secret:latest | FILECHECK | FILENAMEMATCH | application of    | STOP        |
|              |                   |           |               | regexp matched    |             |
|              |                   |           |               | file found in     |             |
|              |                   |           |               | container: file=/ |             |
|              |                   |           |               | root/.ssh/id_rsa  |             |
|              |                   |           |               | regexp=.*/.ssh/id |             |
|              |                   |           |               | _rsa$             |             |
| 9f767c5486f4 | aic-secret:latest | FINAL     | FINAL         |                   | STOP        |
+--------------+-------------------+-----------+---------------+-------------------+-------------+

This simple policy performs just a few checks on your image and in the case of secrets only looks for private SSH keys however this policy can easily be extended to look for any secrets or blacklisted artifacts in your image.

Anchore 1.1 Has Arrived

Posted on February 9, 2017January 5, 2022 by Christian Wiens

We started the week with an exciting announcement about the Anchore Navigator which received a significant update with many new features, the two new features that are proving to be the most popular are the ability submit an image for analysis and the ability to subscribe to receive notifications when an image has been updated. But that’s not the only release that Anchore is announcing this week.

We are proud to announce the 1.1 release of the Anchore’s open source project. The open source engine is at the heart of all of our products – the Navigator, our SaaS service and our on-premise solution. The team at Anchore believes strongly in open source and especially in the need for open source solutions around compliance and governance.

How do you have confidence in a certification test if you don’t know that the test is being performed accurately and without any bias? By building the solution on top of an open source engine with compliance policies that are publicly available, anyone can re-run these tests to verify the results, in short, you can “trust but verify”.

The Most Notable Improvements in the 1.1 Release

Support for Ruby Gems

Anchore now supports detailed scanning for Ruby Gems. All Gems within the container image are reported including their name, version, origin, source, license and location. Anchore’s commercial release now includes a Gem data feed that provides detailed information about Ruby Gems published on the official Gem repository and this information can be used during policy evaluations. For example to check if a Gem comes from the official repository or to report on Ruby Gems that are not up to date. Other policy checks include blacklisting and license checking.

CVE scanning for Alpine Linux

Previously Anchore could report on files and packages within Alpine Linux based images but not report on CVEs. This release adds support for scanning Alpine images and reporting on known CVEs based on the vulnerability data found in Alpine’s security database and within the National Vulnerability Database (NVD) maintained by NIST.

Global Whitelisting

Anchore supports the creation of whitelists on a per-image basis – for example, “exclude CVE-2015-8710 from policy evaluation for image myapp:latest”. The 1.1 release allows a global whitelist to be created allowing organizations to define a curated list of CVEs or other policy checks that are globally excluded during policy evaluation.

Debian CVE scanning

Debian CVE reporting has been updated and will show the binary package that contains the CVE rather than the corresponding source package.

UX and performance

A number of additional improvements have been made to improve user experience – for example simplifying command line options and to improve the performance of scanning.

More details can be found in the changelog on GitHub.

You learn more about our open source release here or contact us using the form below to schedule a 1-on-1 product demonstration.

A Better Way to Navigate Container Registries

Posted on February 6, 2017January 5, 2022 by Christian Wiens

In October 2016 Anchore announced the first release of our commercial product, built on top of our open source container analysis engine. The focus of the open source project and the commercial offering is to deliver tools that perform deep analysis on container images and allow organizations to define policies that govern the deployment of their containers, ensuring that only containers that comply with the organization’s security policy or operational best practices are deployed.

At the same time, we also released the Anchore Navigator which provided a free service to allow users to discover and analyze images on public container registries. At launch, the Navigator included in-depth analysis of all official repositories on DockerHub and 50 of the most popular repositories. Then early in December, we updated the Navigator to add support for basic analysis of all public images on DockerHub allowing users to view basic information such as the image size, layer information, image ID, Digest and creation date.

Today we are announcing a new release of the Navigator that adds a number of powerful new features to this free SaaS service.

Submit Images for Analysis

The first new feature adds the ability for users to submit any public tagged image to Anchore for analysis.

At the top of the preview page for an image, there is a button to submit the image for analysis.
Once submitted this TAG is added to Anchore’s catalog and will be queued up to be downloaded and analyzed. After the first analysis, Anchore will poll the registry for changes and will download new versions of the TAG for analysis whenever the TAG is updated.

Subscriptions

Another powerful new feature is Subscriptions. Users can subscribe to a TAG and will be notified when the TAG is updated. For example, if you use ubuntu:latest as the base image for your containers then when the Ubuntu community push a new ubuntu:latest image to the registry you will receive a notification email from Anchore. Webhook notifications will be added in an upcoming release.

Images can be marked as “favorites” to allow users to quickly access these images.

Organizing Images

A new option has been added on the menu bar for “My Images”

Within the ‘My Images’ page users can view their favorite and subscribed images and quickly see the status of these images – for example, to see when an image was last updated.

Ruby Gems Support

In addition to operating system packages, all files and Node.JS NPMs the Navigator now allows you to see a detailed list of all Ruby GEMs installed in the image, showing details of the packages, including version, license, location and Origin.

Support for Alpine Linux

Anchore Navigator now supports CVE scanning of Alpine Linux images, incorporating security feeds from the Alpine Projects Vulnerabilities database and the National Vulnerabilities Database.

Registry Support

The Navigator has been built to support multiple registries both public and private registries and to analyze images in Docker’s native format and the upcoming Open Containers Initiative (OCI) Image Format. Over the coming months, more registries including private ISV registries will be included within the Navigator’s catalog.

There are more interesting features in development including support for WebHook and Slack notifications, support for deeper analysis of Python libraries and Java Archives along with the ability to analyze private images and define custom policies and whitelists in the commercial Navigator offering.

Comparing Images

Posted on January 10, 2017April 13, 2022 by Christian Wiens

As anyone who has worked in IT support or operations for any period of time will tell you, if you get a call telling you that something stopped working, then the first question you should ask is “what changed?”. This is especially true if the application or server in question has been working well for sometime before.

Keeping track of what changed, or preventing changes from occurring is an important part of IT today, so much so that there is a large ecosystem of vendors and open source projects covering change/release management and monitoring.

Knowing just that something has changed is a good first step but you really need to know the details of what changed. The most common way to do this is to look at the changelog.

Maintaining a changelog for your application or other software project is considered best practice today and it is important to make sure the changelog is well structured and contains all relevant, but notable, information. As one great resource explains “Don’t let your friends dump git logs into CHANGELOGs”.

Operating system vendors typically create release notes that provide a high-level summary of the notable changes in a release, for example in the release notes for CentOS 7.3 and these vendors also include changelogs for individual software packages. For example:

# rpm -q --changelog glibc
* Fri Dec 23 2016 Carlos O'Donell <[email protected]> - 2.24-4
- Auto-sync with upstream release/2.24/master,
 commit e9e69e468039fcd57276f783a16aa771a8e4214e, fixing:
- Shared object unload assert when calling dlclose (#1398370, swbz#11941)
- Fix runtime resolver routines in the presence of AVX512 (swbz#20508)
- Fix writes past the allocated array bounds in execvpe (swbz#20847)
- Fix building with GCC 6.2 on i686 with stack protector.
- Fix building with GCC 7.
- Fix POWER6 memset with recent binutils.
- Fix POWER math test expected failures.
- Fix cancellation in posix_spawn.
- Fix multiarch builds for POWER9.
…

But in the world of containers things aren’t quite so easy. Containers are, by design, opaque.
A user downloads an application container for the application they want, for example, NGINX, and may not know how that container is built, for example, what operating system is used under the covers, let alone what changes were made between releases.

There are no easy ways to perform a “diff” on Docker container images to see what has changed between versions. While there is a docker diff command this command shows what files have changed in a running container but will not show changes between container images. You could also look at the Dockerfile, however, the same Dockerfile used at two different times will likely produce different images since the underlying operating system packages and application files may have been updated.

So today we want to show you how you can compare two container images to see what changes have been made.

For this example, I’ll compare the latest version of the CentOS image with the previously published version.

If you want to visually inspect the latest CentOS image you can do so using Anchore Navigator you can simply search for CentOS and then select the ‘latest’ tag or you can go directly to this link:

Here you can see that this image was last updated on the 15th of December.

I’m going to pull down this image to my local machine by running

# docker pull centos:latest

Running docker images on my local machine will show this latest version of CentOS, however, if I don’t have the previous centos:latest image I need to pull that image from Docker Hub.

While it’s simple to get the current centos:latest image from Docker Hub it’s not quite so easy to find the previous version, however, that’s something that Anchore Navigator can help with. On the overview page of the centos:latest image you’ll see a Previous Image button in the top left, clicking that will take you to the previous version of centos:latest, or you can go directly there using this link.

In the screenshot below you can see that this version is no longer tagged, it’s still available on Docker Hub but no longer has the latest tag or any other tag. It was published on the 2nd of November and then replaced on the 15th of December.

One little known feature of Docker & Docker Hub is the ability to pull an image by its digest.

So you can click the button next to the digest to copy the digest into the clipboard and then run the following command:

docker pull centos@sha256:b2f9d1c0ff5f87a4743104d099a3d561002ac500db1b9bfa02a783a46e0d366c

This will pull down the previous version of centos:latest.

Running docker images --digests centos will show the centos images along with their corresponding digests and IDs.

REPOSITORY     TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
docker.io/centos    latest              sha256:c577af3197aacedf79c5a204cd7f493c8e07ffbce7f88f7600bf19c688c38799   67591570dd29        3 weeks ago         191.8 MB
docker.io/centos                        sha256:b2f9d1c0ff5f87a4743104d099a3d561002ac500db1b9bfa02a783a46e0d366c   0584b3d2cf6d        9 weeks ago         196.5 MB
docker.io/centos                        sha256:2ae0d2c881c7123870114fb9cc7afabd1e31f9888dac8286884f6cf59373ed9b   980e0e4c79ec        4 months ago        196.7 MB
docker.io/centos    7.2.1511            sha256:0d121fa7987c60c3f7ecb8d7347d8e86683018625e44f3864e69b388087a4d0b   feac5e0dfdb2        4 months ago        194.6 MB
docker.io/centos    7.0.1406                                                                                      68c19b8863f0        6 months ago        210.2 MB
docker.io/centos    7.1.1503                                                                                      80d283436f62        6 months ago        212.1 MB

We will now use Anchore to analyze both images.

# anchore analyze --imagetype=none --image=centos:latest

# anchore analyze --imagetype=none --image=0584b3d2cf6d

Now that anchore has analyzed the images we can perform queries on the images.

# anchore query --image=centos:latest show-pkg-diffs 0584b3d2cf6d

This command will show the differences in package manifests between the two images, a portion of that output is included below:

+--------------+-------------------------+------------------+--------------------------+----------------------+--------------------------+
| Image Id     | Repo Tag                | Compare Image Id | Package                  | Input Image Version  | Compare Image Version    |
+--------------+-------------------------+------------------+--------------------------+----------------------+--------------------------+
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | nss-tools                | 3.21.3-2.el7_3       | 3.21.0-9.el7_2           |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | python-urlgrabber        | 3.10-8.el7           | 3.10-7.el7               |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | iputils                  | 20160308-8.el7       | 20121221-7.el7           |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | expat                    | 2.1.0-10.el7_3       | 2.1.0-8.el7              |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | audit-libs               | 2.6.5-3.el7          | 2.4.1-5.el7              |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | gnupg2                   | 2.0.22-4.el7         | 2.0.22-3.el7             |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | xz                       | 5.2.2-1.el7          | 5.1.2-12alpha.el7        |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | nss-sysinit              | 3.21.3-2.el7_3       | 3.21.0-9.el7_2           |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | file-libs                | 5.11-33.el7          | 5.11-31.el7              |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | rpm-build-libs           | 4.11.3-21.el7        | 4.11.3-17.el7            |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | libgcc                   | 4.8.5-11.el7         | 4.8.5-4.el7              |

This default formatting is designed for viewing in the terminal however you can use the --json or --plain command line options to produce output more suited to automated processing.

For example:

# anchore --json query --image=centos:latest show-pkg-diffs 0584b3d2cf6d

Anchore also includes a command to show what files have changed in an image.

# anchore  query --image=centos:latest show-file-diffs 0584b3d2cf6d

+--------------+-------------------------+------------------+-----------------------------------+-----------------------------------+-----------------------------------+
| Image Id     | Repo Tag                | Compare Image Id | File                              | Input Image File Checksum         | Compare Image Checksum            |
+--------------+-------------------------+------------------+-----------------------------------+-----------------------------------+-----------------------------------+
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /usr/bin/signtool                 | d0fd71514d28636fa0afd28f2ce8a04dc | 3094cc4c9f8b507513bd945cad92b2098 |
|              |                         |                  |                                   | a9d837e45895900ce3a293adfec4adb   | b8d6bf84956bbf1adec828690fc48c6   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /var/lib/yum/yumdb/l/8b0fec58c4cb | ec25c418f1f5d51128ddbf924e633b3c5 | NOTINSTALLED                      |
|              |                         |                  | 6f239014f68fff4b4f8681694628-libb | 102649304f1c1a106afccd061f6aa35   |                                   |
|              |                         |                  | lkid-2.23.2-33.el7-x86_64/checksu |                                   |                                   |
|              |                         |                  | m_data                            |                                   |                                   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /usr/bin/sha224sum                | af0e2ff0d30605159cf6d79fc59055b1a | 5c233b844571c856ce9cb7059a88e0cf6 |
|              |                         |                  |                                   | 87fdff577358439844afcbc98ca1acf   | 0ee372d43f1f1cc4a02599b9d3ac8d0   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /usr/lib64/python2.7/lib-         | 1506d2df911351ae57e0c498adfa3faa4 | f0c9c6f0f6b1597624c7bb4cb55d4f2d7 |
|              |                         |                  | dynload/_codecs_kr.so             | 408ca4df07466fafa69a10613b11922   | 9d319697dff3bf6ffb27fac47afc0f7   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /var/lib/yum/yumdb/s/c13227f13b29 | NOTINSTALLED                      | DIRECTORY_OR_OTHER                |
|              |                         |                  | f6866c96d050aebd6098d7a62809-setu |                                   |                                   |
|              |                         |                  | p-2.8.71-6.el7-noarch/checksum_ty |                                   |                                   |
|              |                         |                  | pe                                |                                   |                                   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /usr/share/licenses/device-       | NOTINSTALLED                      | 32b1062f7da84967e7019d01ab805935c |
|              |                         |                  | mapper-libs-1.02.107/COPYING      |                                   | aa7ab7321a7ced0e30ebe75e5df1670   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /usr/lib64/gconv/DEC-MCS.so       | 5d098b7ce2079a621a0f99ae44f959f20 | 764b1597a91a39f799dd3f96051540864 |
|              |                         |                  |                                   | 15a1d64527650f2cc47982a4d9bd3ab   | 9089e7b2154541a630e92fa586c11f9   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /var/lib/yum/yumdb/r/1f4b80c13100 | NOTINSTALLED                      | DIRECTORY_OR_OTHER                |
|              |                         |                  | 951f6f606b7ee0519abe674f0168-rpm- |                                   |                                   |
|              |                         |                  | build-                            |                                   |                                   |
|              |                         |                  | libs-4.11.3-17.el7-x86_64/reason  |                                   |                                   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /usr/lib64/python2.7/symtable.pyc | 16eef0372b200028ae390b22dd1093b00 | 533e494e479e040772edfedfdc70c2923 |
|              |                         |                  |                                   | 772c6011002c8cfb08e01e183a55dfd   | 333ceeac3520f8ef42f170b74317425   |

The challenge with interpreting the output of this command is that nearly 4,000 files have changed since 80 packages have changed so there’s a lot of “noise” since these file changes are expected. We should still look for file changes since files that are not part of an operating system package may be changed, for example, configuration files or application files.

To make this easier the enterprise release of Anchore contains the new command to show the files that are now owned by an operating system package.

# anchore query --image=centos:latest show-non-packaged-files all /

The all parameter specifies that all changes should be displayed. We can use a number such as 2 to specify the depth of directories that are analyzed, for example using 2 would show just the top-level directories that contain changes.

+--------------+-------------------------+------------------------------------------------+
| Image Id     | Repo Tags               | File/Directory Name                            |
+--------------+-------------------------+------------------------------------------------+
| 67591570dd29 | docker.io/centos:latest | /var/log/anaconda/storage.log                  |
| 67591570dd29 | docker.io/centos:latest | /run/systemd/sessions                          |
| 67591570dd29 | docker.io/centos:latest | /run/user                                      |
| 67591570dd29 | docker.io/centos:latest | /tmp/yum.log                                   |
| 67591570dd29 | docker.io/centos:latest | /usr/lib/locale                                |
| 67591570dd29 | docker.io/centos:latest | /tmp/.X11-unix                                 |
| 67591570dd29 | docker.io/centos:latest | /etc/sysconfig/network-scripts                 |
| 67591570dd29 | docker.io/centos:latest | /usr/lib64/p11-kit-trust.so                    |
| 67591570dd29 | docker.io/centos:latest | /var/log/anaconda/ks-script-s0_pQV.log         |
| 67591570dd29 | docker.io/centos:latest | /usr/lib64/pkcs11                              |
| 67591570dd29 | docker.io/centos:latest | /etc/rsyslog.d                                 |
| 67591570dd29 | docker.io/centos:latest | /var/log/anaconda/ifcfg.log                    |
| 67591570dd29 | docker.io/centos:latest | /tmp/ks-script-LRoSA2                          |
| 67591570dd29 | docker.io/centos:latest | /lost+found                                    |
| 67591570dd29 | docker.io/centos:latest | /etc/crypttab                                  |
| 67591570dd29 | docker.io/centos:latest | /run/systemd/machines                          |
| 67591570dd29 | docker.io/centos:latest | /run/log                                       |
| 67591570dd29 | docker.io/centos:latest | /usr/lib/firewalld/ipsets                      |
| 67591570dd29 | docker.io/centos:latest | /etc/alternatives/ld                           |
| 67591570dd29 | docker.io/centos:latest | /etc/group-                                    |
| 67591570dd29 | docker.io/centos:latest | /usr/lib64/fipscheck                           |
| 67591570dd29 | docker.io/centos:latest | /var/lib/alternatives/libnssckbi.so.x86_64     |
| 67591570dd29 | docker.io/centos:latest | /etc/openldap/certs/password                   |
| 67591570dd29 | docker.io/centos:latest | /var/lib/yum/yumdb                             |
| 67591570dd29 | docker.io/centos:latest | /etc/systemd/system/multi-user.target.wants    |
| 67591570dd29 | docker.io/centos:latest | /etc/systemd/system/system-update.target.wants |

And there is a similar command to compare non-packaged files between images which shows the files that are not part of operating system packages that have changed between two images.

# anchore query --image=centos:latest show-non-packaged-files-diff all / 0584b3d2cf6d

When performing this query on your own images you may find a lot of noise caused by temporary files or logs. For example /var/lib/yum may contain data from package installs or updates. Directories can be filtered out using the exclude= option on the command line.

To summarize – you should be able to quickly produce a changelog for a container in 3 simple steps:

Step 1: Analyze the images you wish to compare:

# anchore analyze --imagetype=none --image=myapp:latest
# anchore analyze --imagetype=none --image=myapp:old

Step 2: Run a query to report on the package changes

anchore query --image=centos:latest show-pkg-diffs my app:old

Step 3: Run a query to report on the (non packaged) file changes

# anchore query --image=centos:latest show-non-packaged-files-diff all / myapp:old

These commands will produce human-readable output, complete with tables, however, you can easily add
--json or --plain to produce machine-parsable output.

You can download and install the Anchore open source project now on GitHub or request a demo of Anchore Enterprise.

Hanlon’s Images

Posted on January 5, 2017 by Christian Wiens

Occam’s razor is a well known philosophical principle that’s entered mainstream culture.
While there are many ways to describe this principle the most succinct is:

“The simplest answer is most often correct.”

The lesson behind this razor is that if there are many explanations for a particular phenomenon, then out of the many and often complex alternative explanations the simplest is likely the most likely to be correct.

In philosophy, a razor is a principle that helps you “shave off” unlikely explanations.
I’d like to share with you another razor, one that is less well known but that I have found to be very useful in assessing situations I encounter in day to day life and especially around security.

Hanlon’s Razor states:

“Never attribute to malice that which is adequately explained by stupidity”

Or in other words:

“Don’t assume bad intentions over neglect and misunderstanding.”

Over the last 6 months, we’ve spoken to many organizations about container security and the need to apply governance within their container infrastructure. One question that has come up often is this: “If I’m only using official images or building my own images why do I need to scan?” This is a fair question and before I invoke Hanlon it’s worth a little discussion.

Let’s start with the first point, Official images:
Of the many thousand repositories on DockerHub there are around 140 special repositories that are classified as Official repos. These are a set of curated repos that have been created by an organization or community and submitted to Docker Inc and the community for official review. The official repos are among the most popular images on DockerHub and undergo detailed review before being classified as official including adherence to Dockerfile best practices in creating the image.

Using the Official repos, especially the base operating system images, is a good best practice as it ensures that you are starting off with content from a known source. But care should still be taken in the use of these images. While some official images are updated frequently, many images (especially base OS images) are often only updated on a monthly basis or sometimes even less frequently. A quick way to get an idea of this is to look at the Anchore Navigator and sort by the “Repo Last Updated” column. You’ll notice an “Update Frequency” column on the Navigator which currently displays “Gathering Data”. Anchore’s cloud service continually monitors DockerHub for changes, pulling down and analyzing images as they are updated. We’ve gathered several months’ worth of history and over the coming weeks, we’ll give a visual indication of the frequency of updates within this column. Another common problem that we’ve heard from many organizations is that while a developer may be using the image tagged latest they may not be aware that the latest image has been updated and so the ‘new latest’ image needs to be pulled down from DockerHub.

Regardless of how often an image is updated it should still be scanned for vulnerabilities before deployment as many of the official images, not to mention the tens of thousands of public images, contain exploitable vulnerabilities. So whether you base your containers off an official image from DockerHub, a public image, or build an image from scratch then it’s good practice to ensure that all the latest package updates have been applied to the image.

Scanning and updating the operating system packages is just the first step in ensuring that your images are secure, while this will address common issues such as known vulnerabilities in operating system packages (CVEs) we believe that this is just the tip of the iceberg. It’s possible to have all the latest operating system packages but still have an image that has security vulnerabilities or is otherwise not compliant with your operational, security or business policies. One area that is often overlooked is third party software libraries that are used within your applications such as Node.JS modules pulled from the public NPM or Ruby GEM repositories. A great example of this came at the end of December where a remote code execution vulnerability was reported in the PHPMailer library that’s widely used in many in-house PHP applications as well as common off the shelf applications such as WordPress, Drupal and SugarCRM. While a CVE has been assigned to this vulnerability a simple scan of operating system packages would likely not find this since many developers do not pull in their PHP, Ruby or Node libraries from operating system packages.

Even with the latest operating system packages and with well-written applications using up to date libraries, a container image may be made insecure due to misconfiguration which may be caused by administration or debugging options that are left enabled or by misconfigured encryption or SSL certificates or through enabling unnecessary services within your container image. A great example of this was seen last year at Vine where a security researcher found source code and API keys embedded within the container image.

And Here is Where Hanlon can Help

“Don’t assume bad intentions over neglect and misunderstanding.”

In all likelihood the security or compliance issues that you will encounter within your image won’t be due to malicious intent – where a hacker has embedded malware in a public image that you consume, most of the issues will be caused by mistakes that are made: packages that are not updated, 3rd party libraries that are vulnerable or simple application misconfigurations which is why scanning should be in place for all images no matter the source of the image even if all content was developed in house – you are looking for mistakes as well as for malice.

While image scanning solutions focus on scanning the operating system image for known vulnerabilities Anchore provides a deeper level of analysis into images looking at the operating system, 3rd party libraries, configuration files, etc. With Anchore, organizations can define their own policies that describe their certification needs, covering all aspects of the images, that can be run at any time both on images that they may consume from public sources and on images that are created in house.

One cautionary note:

The Most Notable Improvements in the 1.1 Release

Support for Ruby Gems

CVE scanning for Alpine Linux

Global Whitelisting

Debian CVE scanning

UX and performance