While that may sound like advice your mother gave you after you got into a fight at school we are actually talking about Docker images.
Yesterday we started to notice a lot of activity on our worker nodes on anchore.io which were analyzing a large number of images that were updated on Docker Hub.
The Anchore service monitors Docker Hub looking for changes made to our customer’s private images, official images and thousands of other tags of popular images on Docker Hub.
We poll Docker Hub and when images are updated our workers pull down the new images and perform analysis and policy evaluations. Users can also subscribe to images to get notifications when images they use are updated.
Since yesterday we’ve seen over a thousand images get updated including official OS based images such as Alpine, CentOS, Debian, Oracle, and Ubuntu.
What was odd was that looking at these images we saw no changes in files or package manifests. As part of Anchore’s analysis we look at all the files in the image down to the checksum level and all the package data, this allows us to perform policy checks that go beyond the usual CVE checks that you see with most tools.
We show a brief changelog summary on the overview page for an image, showing how many files and packages were added, removed or changed.
What had us scratching our heads yesterday was the high number of images with no apparent changes. The image metadata, such as ID and Digest were changed but the underlying content was the same.
Digging deeper it appears that while with the actual content of the images has not changed, the manifests have been updated. This seems to have been driven by a change to the bashbrew utility which is used to build official images. Bashbrew now defaults to using the manifest list format which allows for multi-arch images, so even if an image has been built only for a single architecture it will now use the manifest list.
We will continue to dig into this but in the meantime, we’d recommend that you look to see what, if anything, changed in an image before you rebuild all your application images on top of a new base image.
