January 2017

As anyone who has worked in IT support or operations for any period of time will tell you, if you get a call telling you that something stopped working, then the first question you should ask is “what changed?”. This is especially true if the application or server in question has been working well for sometime before.

Keeping track of what changed, or preventing changes from occurring is an important part of IT today, so much so that there is a large ecosystem of vendors and open source projects covering change/release management and monitoring.

Knowing just that something has changed is a good first step but you really need to know the details of what changed. The most common way to do this is to look at the changelog.

Maintaining a changelog for your application or other software project is considered best practice today and it is important to make sure the changelog is well structured and contains all relevant, but notable, information. As one great resource explains “Don’t let your friends dump git logs into CHANGELOGs”.

Operating system vendors typically create release notes that provide a high-level summary of the notable changes in a release, for example in the release notes for CentOS 7.3 and these vendors also include changelogs for individual software packages. For example:

# rpm -q --changelog glibc
* Fri Dec 23 2016 Carlos O'Donell <[email protected]> - 2.24-4
- Auto-sync with upstream release/2.24/master,
 commit e9e69e468039fcd57276f783a16aa771a8e4214e, fixing:
- Shared object unload assert when calling dlclose (#1398370, swbz#11941)
- Fix runtime resolver routines in the presence of AVX512 (swbz#20508)
- Fix writes past the allocated array bounds in execvpe (swbz#20847)
- Fix building with GCC 6.2 on i686 with stack protector.
- Fix building with GCC 7.
- Fix POWER6 memset with recent binutils.
- Fix POWER math test expected failures.
- Fix cancellation in posix_spawn.
- Fix multiarch builds for POWER9.
…

But in the world of containers things aren’t quite so easy. Containers are, by design, opaque.
A user downloads an application container for the application they want, for example, NGINX, and may not know how that container is built, for example, what operating system is used under the covers, let alone what changes were made between releases.

There are no easy ways to perform a “diff” on Docker container images to see what has changed between versions. While there is a docker diff command this command shows what files have changed in a running container but will not show changes between container images. You could also look at the Dockerfile, however, the same Dockerfile used at two different times will likely produce different images since the underlying operating system packages and application files may have been updated.

So today we want to show you how you can compare two container images to see what changes have been made.

For this example, I’ll compare the latest version of the CentOS image with the previously published version.

If you want to visually inspect the latest CentOS image you can do so using Anchore Navigator you can simply search for CentOS and then select the ‘latest’ tag or you can go directly to this link:

Here you can see that this image was last updated on the 15th of December.

I’m going to pull down this image to my local machine by running

# docker pull centos:latest

Running docker images on my local machine will show this latest version of CentOS, however, if I don’t have the previous centos:latest image I need to pull that image from Docker Hub.

While it’s simple to get the current centos:latest image from Docker Hub it’s not quite so easy to find the previous version, however, that’s something that Anchore Navigator can help with. On the overview page of the centos:latest image you’ll see a Previous Image button in the top left, clicking that will take you to the previous version of centos:latest, or you can go directly there using this link.

In the screenshot below you can see that this version is no longer tagged, it’s still available on Docker Hub but no longer has the latest tag or any other tag. It was published on the 2nd of November and then replaced on the 15th of December.

One little known feature of Docker & Docker Hub is the ability to pull an image by its digest.

So you can click the button next to the digest to copy the digest into the clipboard and then run the following command:

docker pull centos@sha256:b2f9d1c0ff5f87a4743104d099a3d561002ac500db1b9bfa02a783a46e0d366c

This will pull down the previous version of centos:latest.

Running docker images --digests centos will show the centos images along with their corresponding digests and IDs.

REPOSITORY     TAG                 DIGEST                                                                    IMAGE ID            CREATED             SIZE
docker.io/centos    latest              sha256:c577af3197aacedf79c5a204cd7f493c8e07ffbce7f88f7600bf19c688c38799   67591570dd29        3 weeks ago         191.8 MB
docker.io/centos                        sha256:b2f9d1c0ff5f87a4743104d099a3d561002ac500db1b9bfa02a783a46e0d366c   0584b3d2cf6d        9 weeks ago         196.5 MB
docker.io/centos                        sha256:2ae0d2c881c7123870114fb9cc7afabd1e31f9888dac8286884f6cf59373ed9b   980e0e4c79ec        4 months ago        196.7 MB
docker.io/centos    7.2.1511            sha256:0d121fa7987c60c3f7ecb8d7347d8e86683018625e44f3864e69b388087a4d0b   feac5e0dfdb2        4 months ago        194.6 MB
docker.io/centos    7.0.1406                                                                                      68c19b8863f0        6 months ago        210.2 MB
docker.io/centos    7.1.1503                                                                                      80d283436f62        6 months ago        212.1 MB

We will now use Anchore to analyze both images.

# anchore analyze --imagetype=none --image=centos:latest

# anchore analyze --imagetype=none --image=0584b3d2cf6d

Now that anchore has analyzed the images we can perform queries on the images.

# anchore query --image=centos:latest show-pkg-diffs 0584b3d2cf6d

This command will show the differences in package manifests between the two images, a portion of that output is included below:

+--------------+-------------------------+------------------+--------------------------+----------------------+--------------------------+
| Image Id     | Repo Tag                | Compare Image Id | Package                  | Input Image Version  | Compare Image Version    |
+--------------+-------------------------+------------------+--------------------------+----------------------+--------------------------+
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | nss-tools                | 3.21.3-2.el7_3       | 3.21.0-9.el7_2           |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | python-urlgrabber        | 3.10-8.el7           | 3.10-7.el7               |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | iputils                  | 20160308-8.el7       | 20121221-7.el7           |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | expat                    | 2.1.0-10.el7_3       | 2.1.0-8.el7              |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | audit-libs               | 2.6.5-3.el7          | 2.4.1-5.el7              |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | gnupg2                   | 2.0.22-4.el7         | 2.0.22-3.el7             |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | xz                       | 5.2.2-1.el7          | 5.1.2-12alpha.el7        |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | nss-sysinit              | 3.21.3-2.el7_3       | 3.21.0-9.el7_2           |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | file-libs                | 5.11-33.el7          | 5.11-31.el7              |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | rpm-build-libs           | 4.11.3-21.el7        | 4.11.3-17.el7            |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | libgcc                   | 4.8.5-11.el7         | 4.8.5-4.el7              |

This default formatting is designed for viewing in the terminal however you can use the --json or --plain command line options to produce output more suited to automated processing.

For example:

# anchore --json query --image=centos:latest show-pkg-diffs 0584b3d2cf6d

Anchore also includes a command to show what files have changed in an image.

# anchore  query --image=centos:latest show-file-diffs 0584b3d2cf6d

+--------------+-------------------------+------------------+-----------------------------------+-----------------------------------+-----------------------------------+
| Image Id     | Repo Tag                | Compare Image Id | File                              | Input Image File Checksum         | Compare Image Checksum            |
+--------------+-------------------------+------------------+-----------------------------------+-----------------------------------+-----------------------------------+
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /usr/bin/signtool                 | d0fd71514d28636fa0afd28f2ce8a04dc | 3094cc4c9f8b507513bd945cad92b2098 |
|              |                         |                  |                                   | a9d837e45895900ce3a293adfec4adb   | b8d6bf84956bbf1adec828690fc48c6   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /var/lib/yum/yumdb/l/8b0fec58c4cb | ec25c418f1f5d51128ddbf924e633b3c5 | NOTINSTALLED                      |
|              |                         |                  | 6f239014f68fff4b4f8681694628-libb | 102649304f1c1a106afccd061f6aa35   |                                   |
|              |                         |                  | lkid-2.23.2-33.el7-x86_64/checksu |                                   |                                   |
|              |                         |                  | m_data                            |                                   |                                   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /usr/bin/sha224sum                | af0e2ff0d30605159cf6d79fc59055b1a | 5c233b844571c856ce9cb7059a88e0cf6 |
|              |                         |                  |                                   | 87fdff577358439844afcbc98ca1acf   | 0ee372d43f1f1cc4a02599b9d3ac8d0   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /usr/lib64/python2.7/lib-         | 1506d2df911351ae57e0c498adfa3faa4 | f0c9c6f0f6b1597624c7bb4cb55d4f2d7 |
|              |                         |                  | dynload/_codecs_kr.so             | 408ca4df07466fafa69a10613b11922   | 9d319697dff3bf6ffb27fac47afc0f7   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /var/lib/yum/yumdb/s/c13227f13b29 | NOTINSTALLED                      | DIRECTORY_OR_OTHER                |
|              |                         |                  | f6866c96d050aebd6098d7a62809-setu |                                   |                                   |
|              |                         |                  | p-2.8.71-6.el7-noarch/checksum_ty |                                   |                                   |
|              |                         |                  | pe                                |                                   |                                   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /usr/share/licenses/device-       | NOTINSTALLED                      | 32b1062f7da84967e7019d01ab805935c |
|              |                         |                  | mapper-libs-1.02.107/COPYING      |                                   | aa7ab7321a7ced0e30ebe75e5df1670   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /usr/lib64/gconv/DEC-MCS.so       | 5d098b7ce2079a621a0f99ae44f959f20 | 764b1597a91a39f799dd3f96051540864 |
|              |                         |                  |                                   | 15a1d64527650f2cc47982a4d9bd3ab   | 9089e7b2154541a630e92fa586c11f9   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /var/lib/yum/yumdb/r/1f4b80c13100 | NOTINSTALLED                      | DIRECTORY_OR_OTHER                |
|              |                         |                  | 951f6f606b7ee0519abe674f0168-rpm- |                                   |                                   |
|              |                         |                  | build-                            |                                   |                                   |
|              |                         |                  | libs-4.11.3-17.el7-x86_64/reason  |                                   |                                   |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d     | /usr/lib64/python2.7/symtable.pyc | 16eef0372b200028ae390b22dd1093b00 | 533e494e479e040772edfedfdc70c2923 |
|              |                         |                  |                                   | 772c6011002c8cfb08e01e183a55dfd   | 333ceeac3520f8ef42f170b74317425   |

The challenge with interpreting the output of this command is that nearly 4,000 files have changed since 80 packages have changed so there’s a lot of “noise” since these file changes are expected. We should still look for file changes since files that are not part of an operating system package may be changed, for example, configuration files or application files.

To make this easier the enterprise release of Anchore contains the new command to show the files that are now owned by an operating system package.

# anchore query --image=centos:latest show-non-packaged-files all /

The all parameter specifies that all changes should be displayed. We can use a number such as 2 to specify the depth of directories that are analyzed, for example using 2 would show just the top-level directories that contain changes.

+--------------+-------------------------+------------------------------------------------+
| Image Id     | Repo Tags               | File/Directory Name                            |
+--------------+-------------------------+------------------------------------------------+
| 67591570dd29 | docker.io/centos:latest | /var/log/anaconda/storage.log                  |
| 67591570dd29 | docker.io/centos:latest | /run/systemd/sessions                          |
| 67591570dd29 | docker.io/centos:latest | /run/user                                      |
| 67591570dd29 | docker.io/centos:latest | /tmp/yum.log                                   |
| 67591570dd29 | docker.io/centos:latest | /usr/lib/locale                                |
| 67591570dd29 | docker.io/centos:latest | /tmp/.X11-unix                                 |
| 67591570dd29 | docker.io/centos:latest | /etc/sysconfig/network-scripts                 |
| 67591570dd29 | docker.io/centos:latest | /usr/lib64/p11-kit-trust.so                    |
| 67591570dd29 | docker.io/centos:latest | /var/log/anaconda/ks-script-s0_pQV.log         |
| 67591570dd29 | docker.io/centos:latest | /usr/lib64/pkcs11                              |
| 67591570dd29 | docker.io/centos:latest | /etc/rsyslog.d                                 |
| 67591570dd29 | docker.io/centos:latest | /var/log/anaconda/ifcfg.log                    |
| 67591570dd29 | docker.io/centos:latest | /tmp/ks-script-LRoSA2                          |
| 67591570dd29 | docker.io/centos:latest | /lost+found                                    |
| 67591570dd29 | docker.io/centos:latest | /etc/crypttab                                  |
| 67591570dd29 | docker.io/centos:latest | /run/systemd/machines                          |
| 67591570dd29 | docker.io/centos:latest | /run/log                                       |
| 67591570dd29 | docker.io/centos:latest | /usr/lib/firewalld/ipsets                      |
| 67591570dd29 | docker.io/centos:latest | /etc/alternatives/ld                           |
| 67591570dd29 | docker.io/centos:latest | /etc/group-                                    |
| 67591570dd29 | docker.io/centos:latest | /usr/lib64/fipscheck                           |
| 67591570dd29 | docker.io/centos:latest | /var/lib/alternatives/libnssckbi.so.x86_64     |
| 67591570dd29 | docker.io/centos:latest | /etc/openldap/certs/password                   |
| 67591570dd29 | docker.io/centos:latest | /var/lib/yum/yumdb                             |
| 67591570dd29 | docker.io/centos:latest | /etc/systemd/system/multi-user.target.wants    |
| 67591570dd29 | docker.io/centos:latest | /etc/systemd/system/system-update.target.wants |

And there is a similar command to compare non-packaged files between images which shows the files that are not part of operating system packages that have changed between two images.

# anchore query --image=centos:latest show-non-packaged-files-diff all / 0584b3d2cf6d

When performing this query on your own images you may find a lot of noise caused by temporary files or logs. For example /var/lib/yum may contain data from package installs or updates. Directories can be filtered out using the exclude= option on the command line.

To summarize – you should be able to quickly produce a changelog for a container in 3 simple steps:

Step 1: Analyze the images you wish to compare:

# anchore analyze --imagetype=none --image=myapp:latest
# anchore analyze --imagetype=none --image=myapp:old

Step 2: Run a query to report on the package changes

anchore query --image=centos:latest show-pkg-diffs my app:old

Step 3: Run a query to report on the (non packaged) file changes

# anchore query --image=centos:latest show-non-packaged-files-diff all / myapp:old

These commands will produce human-readable output, complete with tables, however, you can easily add
--json or --plain to produce machine-parsable output.

You can download and install the Anchore open source project now on GitHub or request a demo of Anchore Enterprise.

Occam’s razor is a well known philosophical principle that’s entered mainstream culture.
While there are many ways to describe this principle the most succinct is:

“The simplest answer is most often correct.”

The lesson behind this razor is that if there are many explanations for a particular phenomenon, then out of the many and often complex alternative explanations the simplest is likely the most likely to be correct.

In philosophy, a razor is a principle that helps you “shave off” unlikely explanations.
I’d like to share with you another razor, one that is less well known but that I have found to be very useful in assessing situations I encounter in day to day life and especially around security.

Hanlon’s Razor states:

“Never attribute to malice that which is adequately explained by stupidity”

Or in other words:

“Don’t assume bad intentions over neglect and misunderstanding.”

Over the last 6 months, we’ve spoken to many organizations about container security and the need to apply governance within their container infrastructure. One question that has come up often is this: “If I’m only using official images or building my own images why do I need to scan?” This is a fair question and before I invoke Hanlon it’s worth a little discussion.

Let’s start with the first point, Official images:
Of the many thousand repositories on DockerHub there are around 140 special repositories that are classified as Official repos. These are a set of curated repos that have been created by an organization or community and submitted to Docker Inc and the community for official review. The official repos are among the most popular images on DockerHub and undergo detailed review before being classified as official including adherence to Dockerfile best practices in creating the image.

Using the Official repos, especially the base operating system images, is a good best practice as it ensures that you are starting off with content from a known source. But care should still be taken in the use of these images. While some official images are updated frequently, many images (especially base OS images) are often only updated on a monthly basis or sometimes even less frequently. A quick way to get an idea of this is to look at the Anchore Navigator and sort by the “Repo Last Updated” column. You’ll notice an “Update Frequency” column on the Navigator which currently displays “Gathering Data”. Anchore’s cloud service continually monitors DockerHub for changes, pulling down and analyzing images as they are updated. We’ve gathered several months’ worth of history and over the coming weeks, we’ll give a visual indication of the frequency of updates within this column. Another common problem that we’ve heard from many organizations is that while a developer may be using the image tagged latest they may not be aware that the latest image has been updated and so the ‘new latest’ image needs to be pulled down from DockerHub.

Regardless of how often an image is updated it should still be scanned for vulnerabilities before deployment as many of the official images, not to mention the tens of thousands of public images, contain exploitable vulnerabilities. So whether you base your containers off an official image from DockerHub, a public image, or build an image from scratch then it’s good practice to ensure that all the latest package updates have been applied to the image.

Scanning and updating the operating system packages is just the first step in ensuring that your images are secure, while this will address common issues such as known vulnerabilities in operating system packages (CVEs) we believe that this is just the tip of the iceberg. It’s possible to have all the latest operating system packages but still have an image that has security vulnerabilities or is otherwise not compliant with your operational, security or business policies. One area that is often overlooked is third party software libraries that are used within your applications such as Node.JS modules pulled from the public NPM or Ruby GEM repositories. A great example of this came at the end of December where a remote code execution vulnerability was reported in the PHPMailer library that’s widely used in many in-house PHP applications as well as common off the shelf applications such as WordPress, Drupal and SugarCRM. While a CVE has been assigned to this vulnerability a simple scan of operating system packages would likely not find this since many developers do not pull in their PHP, Ruby or Node libraries from operating system packages.

Even with the latest operating system packages and with well-written applications using up to date libraries, a container image may be made insecure due to misconfiguration which may be caused by administration or debugging options that are left enabled or by misconfigured encryption or SSL certificates or through enabling unnecessary services within your container image. A great example of this was seen last year at Vine where a security researcher found source code and API keys embedded within the container image.

And Here is Where Hanlon can Help

“Don’t assume bad intentions over neglect and misunderstanding.”

In all likelihood the security or compliance issues that you will encounter within your image won’t be due to malicious intent – where a hacker has embedded malware in a public image that you consume, most of the issues will be caused by mistakes that are made: packages that are not updated, 3rd party libraries that are vulnerable or simple application misconfigurations which is why scanning should be in place for all images no matter the source of the image even if all content was developed in house – you are looking for mistakes as well as for malice.

While image scanning solutions focus on scanning the operating system image for known vulnerabilities Anchore provides a deeper level of analysis into images looking at the operating system, 3rd party libraries, configuration files, etc. With Anchore, organizations can define their own policies that describe their certification needs, covering all aspects of the images, that can be run at any time both on images that they may consume from public sources and on images that are created in house.

Month: January 2017

Comparing Images

Hanlon’s Images

And Here is Where Hanlon can Help