As anyone who has worked in IT support or operations for any period of time will tell you, if you get a call telling you that something stopped working, then the first question you should ask is “what changed?”. This is especially true if the application or server in question has been working well for sometime before.
Keeping track of what changed, or preventing changes from occurring is an important part of IT today, so much so that there is a large ecosystem of vendors and open source projects covering change/release management and monitoring.
Knowing just that something has changed is a good first step but you really need to know the details of what changed. The most common way to do this is to look at the changelog.
Maintaining a changelog for your application or other software project is considered best practice today and it is important to make sure the changelog is well structured and contains all relevant, but notable, information. As one great resource explains “Don’t let your friends dump git logs into CHANGELOGs”.
Operating system vendors typically create release notes that provide a high-level summary of the notable changes in a release, for example in the release notes for CentOS 7.3 and these vendors also include changelogs for individual software packages. For example:
# rpm -q --changelog glibc
* Fri Dec 23 2016 Carlos O'Donell <[email protected]> - 2.24-4
- Auto-sync with upstream release/2.24/master,
commit e9e69e468039fcd57276f783a16aa771a8e4214e, fixing:
- Shared object unload assert when calling dlclose (#1398370, swbz#11941)
- Fix runtime resolver routines in the presence of AVX512 (swbz#20508)
- Fix writes past the allocated array bounds in execvpe (swbz#20847)
- Fix building with GCC 6.2 on i686 with stack protector.
- Fix building with GCC 7.
- Fix POWER6 memset with recent binutils.
- Fix POWER math test expected failures.
- Fix cancellation in posix_spawn.
- Fix multiarch builds for POWER9.
…
But in the world of containers things aren’t quite so easy. Containers are, by design, opaque.
A user downloads an application container for the application they want, for example, NGINX, and may not know how that container is built, for example, what operating system is used under the covers, let alone what changes were made between releases.
There are no easy ways to perform a “diff” on Docker container images to see what has changed between versions. While there is a docker diff command this command shows what files have changed in a running container but will not show changes between container images. You could also look at the Dockerfile, however, the same Dockerfile used at two different times will likely produce different images since the underlying operating system packages and application files may have been updated.
So today we want to show you how you can compare two container images to see what changes have been made.
For this example, I’ll compare the latest version of the CentOS image with the previously published version.
If you want to visually inspect the latest CentOS image you can do so using Anchore Navigator you can simply search for CentOS and then select the ‘latest’ tag or you can go directly to this link:
Here you can see that this image was last updated on the 15th of December.
I’m going to pull down this image to my local machine by running
# docker pull centos:latestRunning docker images on my local machine will show this latest version of CentOS, however, if I don’t have the previous centos:latest image I need to pull that image from Docker Hub.
While it’s simple to get the current centos:latest image from Docker Hub it’s not quite so easy to find the previous version, however, that’s something that Anchore Navigator can help with. On the overview page of the centos:latest image you’ll see a Previous Image button in the top left, clicking that will take you to the previous version of centos:latest, or you can go directly there using this link.
In the screenshot below you can see that this version is no longer tagged, it’s still available on Docker Hub but no longer has the latest tag or any other tag. It was published on the 2nd of November and then replaced on the 15th of December.
One little known feature of Docker & Docker Hub is the ability to pull an image by its digest.
So you can click the 
docker pull centos@sha256:b2f9d1c0ff5f87a4743104d099a3d561002ac500db1b9bfa02a783a46e0d366cThis will pull down the previous version of centos:latest.
Running docker images --digests centos will show the centos images along with their corresponding digests and IDs.
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
docker.io/centos latest sha256:c577af3197aacedf79c5a204cd7f493c8e07ffbce7f88f7600bf19c688c38799 67591570dd29 3 weeks ago 191.8 MB
docker.io/centos sha256:b2f9d1c0ff5f87a4743104d099a3d561002ac500db1b9bfa02a783a46e0d366c 0584b3d2cf6d 9 weeks ago 196.5 MB
docker.io/centos sha256:2ae0d2c881c7123870114fb9cc7afabd1e31f9888dac8286884f6cf59373ed9b 980e0e4c79ec 4 months ago 196.7 MB
docker.io/centos 7.2.1511 sha256:0d121fa7987c60c3f7ecb8d7347d8e86683018625e44f3864e69b388087a4d0b feac5e0dfdb2 4 months ago 194.6 MB
docker.io/centos 7.0.1406 68c19b8863f0 6 months ago 210.2 MB
docker.io/centos 7.1.1503 80d283436f62 6 months ago 212.1 MB
We will now use Anchore to analyze both images.
# anchore analyze --imagetype=none --image=centos:latest# anchore analyze --imagetype=none --image=0584b3d2cf6dNow that anchore has analyzed the images we can perform queries on the images.
# anchore query --image=centos:latest show-pkg-diffs 0584b3d2cf6dThis command will show the differences in package manifests between the two images, a portion of that output is included below:
+--------------+-------------------------+------------------+--------------------------+----------------------+--------------------------+
| Image Id | Repo Tag | Compare Image Id | Package | Input Image Version | Compare Image Version |
+--------------+-------------------------+------------------+--------------------------+----------------------+--------------------------+
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | nss-tools | 3.21.3-2.el7_3 | 3.21.0-9.el7_2 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | python-urlgrabber | 3.10-8.el7 | 3.10-7.el7 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | iputils | 20160308-8.el7 | 20121221-7.el7 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | expat | 2.1.0-10.el7_3 | 2.1.0-8.el7 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | audit-libs | 2.6.5-3.el7 | 2.4.1-5.el7 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | gnupg2 | 2.0.22-4.el7 | 2.0.22-3.el7 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | xz | 5.2.2-1.el7 | 5.1.2-12alpha.el7 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | nss-sysinit | 3.21.3-2.el7_3 | 3.21.0-9.el7_2 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | file-libs | 5.11-33.el7 | 5.11-31.el7 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | rpm-build-libs | 4.11.3-21.el7 | 4.11.3-17.el7 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | libgcc | 4.8.5-11.el7 | 4.8.5-4.el7 |
This default formatting is designed for viewing in the terminal however you can use the --json or --plain command line options to produce output more suited to automated processing.
For example:
# anchore --json query --image=centos:latest show-pkg-diffs 0584b3d2cf6dAnchore also includes a command to show what files have changed in an image.
# anchore query --image=centos:latest show-file-diffs 0584b3d2cf6d+--------------+-------------------------+------------------+-----------------------------------+-----------------------------------+-----------------------------------+
| Image Id | Repo Tag | Compare Image Id | File | Input Image File Checksum | Compare Image Checksum |
+--------------+-------------------------+------------------+-----------------------------------+-----------------------------------+-----------------------------------+
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | /usr/bin/signtool | d0fd71514d28636fa0afd28f2ce8a04dc | 3094cc4c9f8b507513bd945cad92b2098 |
| | | | | a9d837e45895900ce3a293adfec4adb | b8d6bf84956bbf1adec828690fc48c6 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | /var/lib/yum/yumdb/l/8b0fec58c4cb | ec25c418f1f5d51128ddbf924e633b3c5 | NOTINSTALLED |
| | | | 6f239014f68fff4b4f8681694628-libb | 102649304f1c1a106afccd061f6aa35 | |
| | | | lkid-2.23.2-33.el7-x86_64/checksu | | |
| | | | m_data | | |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | /usr/bin/sha224sum | af0e2ff0d30605159cf6d79fc59055b1a | 5c233b844571c856ce9cb7059a88e0cf6 |
| | | | | 87fdff577358439844afcbc98ca1acf | 0ee372d43f1f1cc4a02599b9d3ac8d0 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | /usr/lib64/python2.7/lib- | 1506d2df911351ae57e0c498adfa3faa4 | f0c9c6f0f6b1597624c7bb4cb55d4f2d7 |
| | | | dynload/_codecs_kr.so | 408ca4df07466fafa69a10613b11922 | 9d319697dff3bf6ffb27fac47afc0f7 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | /var/lib/yum/yumdb/s/c13227f13b29 | NOTINSTALLED | DIRECTORY_OR_OTHER |
| | | | f6866c96d050aebd6098d7a62809-setu | | |
| | | | p-2.8.71-6.el7-noarch/checksum_ty | | |
| | | | pe | | |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | /usr/share/licenses/device- | NOTINSTALLED | 32b1062f7da84967e7019d01ab805935c |
| | | | mapper-libs-1.02.107/COPYING | | aa7ab7321a7ced0e30ebe75e5df1670 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | /usr/lib64/gconv/DEC-MCS.so | 5d098b7ce2079a621a0f99ae44f959f20 | 764b1597a91a39f799dd3f96051540864 |
| | | | | 15a1d64527650f2cc47982a4d9bd3ab | 9089e7b2154541a630e92fa586c11f9 |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | /var/lib/yum/yumdb/r/1f4b80c13100 | NOTINSTALLED | DIRECTORY_OR_OTHER |
| | | | 951f6f606b7ee0519abe674f0168-rpm- | | |
| | | | build- | | |
| | | | libs-4.11.3-17.el7-x86_64/reason | | |
| 67591570dd29 | docker.io/centos:latest | 0584b3d2cf6d | /usr/lib64/python2.7/symtable.pyc | 16eef0372b200028ae390b22dd1093b00 | 533e494e479e040772edfedfdc70c2923 |
| | | | | 772c6011002c8cfb08e01e183a55dfd | 333ceeac3520f8ef42f170b74317425 |
The challenge with interpreting the output of this command is that nearly 4,000 files have changed since 80 packages have changed so there's a lot of "noise" since these file changes are expected. We should still look for file changes since files that are not part of an operating system package may be changed, for example, configuration files or application files.
To make this easier the enterprise release of Anchore contains the new command to show the files that are now owned by an operating system package.
# anchore query --image=centos:latest show-non-packaged-files all /The all parameter specifies that all changes should be displayed. We can use a number such as 2 to specify the depth of directories that are analyzed, for example using 2 would show just the top-level directories that contain changes.
+--------------+-------------------------+------------------------------------------------+
| Image Id | Repo Tags | File/Directory Name |
+--------------+-------------------------+------------------------------------------------+
| 67591570dd29 | docker.io/centos:latest | /var/log/anaconda/storage.log |
| 67591570dd29 | docker.io/centos:latest | /run/systemd/sessions |
| 67591570dd29 | docker.io/centos:latest | /run/user |
| 67591570dd29 | docker.io/centos:latest | /tmp/yum.log |
| 67591570dd29 | docker.io/centos:latest | /usr/lib/locale |
| 67591570dd29 | docker.io/centos:latest | /tmp/.X11-unix |
| 67591570dd29 | docker.io/centos:latest | /etc/sysconfig/network-scripts |
| 67591570dd29 | docker.io/centos:latest | /usr/lib64/p11-kit-trust.so |
| 67591570dd29 | docker.io/centos:latest | /var/log/anaconda/ks-script-s0_pQV.log |
| 67591570dd29 | docker.io/centos:latest | /usr/lib64/pkcs11 |
| 67591570dd29 | docker.io/centos:latest | /etc/rsyslog.d |
| 67591570dd29 | docker.io/centos:latest | /var/log/anaconda/ifcfg.log |
| 67591570dd29 | docker.io/centos:latest | /tmp/ks-script-LRoSA2 |
| 67591570dd29 | docker.io/centos:latest | /lost+found |
| 67591570dd29 | docker.io/centos:latest | /etc/crypttab |
| 67591570dd29 | docker.io/centos:latest | /run/systemd/machines |
| 67591570dd29 | docker.io/centos:latest | /run/log |
| 67591570dd29 | docker.io/centos:latest | /usr/lib/firewalld/ipsets |
| 67591570dd29 | docker.io/centos:latest | /etc/alternatives/ld |
| 67591570dd29 | docker.io/centos:latest | /etc/group- |
| 67591570dd29 | docker.io/centos:latest | /usr/lib64/fipscheck |
| 67591570dd29 | docker.io/centos:latest | /var/lib/alternatives/libnssckbi.so.x86_64 |
| 67591570dd29 | docker.io/centos:latest | /etc/openldap/certs/password |
| 67591570dd29 | docker.io/centos:latest | /var/lib/yum/yumdb |
| 67591570dd29 | docker.io/centos:latest | /etc/systemd/system/multi-user.target.wants |
| 67591570dd29 | docker.io/centos:latest | /etc/systemd/system/system-update.target.wants |
And there is a similar command to compare non-packaged files between images which shows the files that are not part of operating system packages that have changed between two images.
# anchore query --image=centos:latest show-non-packaged-files-diff all / 0584b3d2cf6dWhen performing this query on your own images you may find a lot of noise caused by temporary files or logs. For example /var/lib/yum may contain data from package installs or updates. Directories can be filtered out using the exclude= option on the command line.
To summarize - you should be able to quickly produce a changelog for a container in 3 simple steps:
Step 1: Analyze the images you wish to compare:
# anchore analyze --imagetype=none --image=myapp:latest
# anchore analyze --imagetype=none --image=myapp:oldStep 2: Run a query to report on the package changes
anchore query --image=centos:latest show-pkg-diffs my app:old Step 3: Run a query to report on the (non packaged) file changes
# anchore query --image=centos:latest show-non-packaged-files-diff all / myapp:oldThese commands will produce human-readable output, complete with tables, however, you can easily add
--json or --plain to produce machine-parsable output.
You can download and install the Anchore open source project now on GitHub or request a demo of Anchore Enterprise.
