Microservices vs. MicroVM’s

by | Mar 16, 2017

At Anchore we spend a whole lot of time looking at container images to provide detailed analysis and certification. Most of the discussions we hear in the industry around image analysis focus on CVE scanning: how many CVEs are in an image, what severity, etc. As we’ve mentioned before, we see CVE scanning as just the tip of the iceberg and that it’s possible to have all the latest operating system packages but still have an image that has security vulnerabilities or is otherwise not compliant with your operational, security or business policies.

There is another common issue in the tens of thousands of images that we’ve analyzed which we feel is more fundamental. As an industry we are moving to an architecture based on microservices and containers are really the key to enabling this. While the containers we’ve seen are often designed to run microservices, I’d argue that the majority of containers we see (both on DockerHub as well as our customer’s private images) are more like MicroVMs than Microservices. These images typically have a hundred or more packages and several thousand files.

In most cases the images are general purpose operating system images and differ only from their virtual machine brethren by not having a kernel installed. There has been much debate in the industry about image size and how smaller is better, allowing images to be rapidly deployed over the network. Others argue that size doesn’t matter and that the layered nature of Docker’s image format and caching largely mitigates this issue, but looking just at the size of the image doesn’t give you the complete picture.

While it’s certainly an important point to consider the real concern should be not the size of the image, but the content of the images.

Let’s take Alpine as an example. Let’s use the Anchore Navigator to view the contents and select the files tab to drill down further. Filtering this list to show the files in /bin highlights just how many executables are in the image. In a microservice, why does my image need utilities for process or file management? These come from having the busybox package in the image. While that may certainly be useful in some use cases I’d argue that having these kind of binaries in an image that never directly calls them is an accident waiting to happen. I don’t mean to pick on Alpine which weighs in at 4MB (twenty five times smaller than most base operating system images and certainly has less attack surface) but the point you must consider is that you must ensure that every artifact in your image serves a purpose and goes through some form of quality control to ensure that the final image is secure and meets your operational best practises.

Last month Oracle released a slimmed down Oracle Linux image which reduced the footprint down from 225MB to 114MB, you can read our analysis here, this week Red Hat upped the ante when they announced a slimmed down Red Hat Enterprise Linux Atomic Base Image.

The new RHEL image weighs in at 75MB, compared to 192MB for the standard RHEL image. In this image Red Hat has removed a number of packages that are deemed not necessary for container deployments, two of the most interesting removals are systemd and Python. Traditionally all RHEL installs have included python since the YUM package manager is written in Python. To get around this Red Hat has created a new mini package manager called microdnf. While microdnf is not as functional as YUM or as DNF, the next generation package manager for RHEL based distributions, it does just what is needed: install, remove and update packages.

I wanted to look at what else changed in the image so I pulled the RHEL Atomic image from Red Hat’s registry. If you don’t have access to the RHEL registry, you can take a look at the analysis of the image using the Anchore navigator here:

Note: This image is not available publicly on DockerHub.

For the rest of the analysis I’m going to use Anchore’s command line tools which you can install from here.

First I need to analyze the image.

# anchore analyze --image=registry.access.redhat.com/rhel7-atomic

I’ve already analyzed the standard rhel 7 image so now I want to run a query to compare the packages installed in the RHEL Atomic image with the standard RHEL image using the show-pkg-diffs query.

# anchore query --image=registry.access.redhat.com/rhel7 show-pkg-diffs registry.access.redhat.com/rhel7-atomic

 

Package RHEL 7 RHEL Atomic
python-chardet 2.2.1-1.el7_1 Not Installed
librhsm Not Installed 0.0.1-1.el7
yum-plugin-ovl 1.1.31-40.el7 Not Installed
libuser 0.60-7.el7_1 Not Installed
json-glib Not Installed 1.0.2-1.el7
python-urlgrabber 3.10-8.el7 Not Installed
libblkid 2.23.2-33.el7 Not Installed
audit-libs 2.6.5-3.el7_3.1 Not Installed
libsolv Not Installed 0.6.20-5.el7
xz 5.2.2-1.el7 Not Installed
file-libs 5.11-33.el7 Not Installed
rpm-build-libs 4.11.3-21.el7 Not Installed
python-libs 2.7.5-48.el7 Not Installed
qrencode-libs 3.4.1-3.el7 Not Installed
gdbm 1.10-8.el7 Not Installed
cryptsetup-libs 1.7.2-1.el7 Not Installed
dbus-libs 1.6.12-17.el7 Not Installed
tar 1.26-31.el7 Not Installed
dbus-glib 0.100-7.el7 Not Installed
cracklib-dicts 2.9.0-11.el7 Not Installed
kmod 20-9.el7 Not Installed
systemd 219-30.el7_3.7 Not Installed
subscription-manager 1.17.15-1.el7 Not Installed
libpwquality 1.2.3-4.el7 Not Installed
pygpgme 0.3-9.el7 Not Installed
python-dmidecode 3.10.13-11.el7 Not Installed
pyliblzma 0.5.3-11.el7 Not Installed
device-mapper 1.02.135-1.el7_3.3 Not Installed
kmod-libs 20-9.el7 Not Installed
shadow-utils 4.1.5.1-24.el7 Not Installed
python-pycurl 7.19.0-19.el7 Not Installed
libcap-ng 0.7.5-4.el7 Not Installed
python-rhsm-certificates 1.17.9-1.el7 Not Installed
kpartx 0.4.9-99.el7_3.1 Not Installed
python-iniparse 0.4-9.el7 Not Installed
microdnf Not Installed 2-3.el7.1.1
pam 1.1.8-18.el7 Not Installed
cracklib 2.9.0-11.el7 Not Installed
procps-ng 3.3.10-10.el7 Not Installed
pyxattr 0.5.1-5.el7 Not Installed
vim-minimal 7.4.160-1.el7_3.1 Not Installed
python 2.7.5-48.el7 Not Installed
python-rhsm 1.17.9-1.el7 Not Installed
python-ethtool 0.8-5.el7 Not Installed
cpio 2.11-24.el7 Not Installed
libutempter 1.1.6-4.el7 Not Installed
device-mapper-libs 1.02.135-1.el7_3.3 Not Installed
systemd-libs 219-30.el7_3.7 Not Installed
dmidecode 3.0-2.el7 Not Installed
m2crypto 0.21.1-17.el7 Not Installed
hardlink 1.0-19.el7 Not Installed
rpm-python 4.11.3-21.el7 Not Installed
yum-utils 1.1.31-40.el7 Not Installed
dbus-python 1.1.1-9.el7 Not Installed
python-dateutil 1.5-7.el7 Not Installed
librepo Not Installed 1.7.16-1.el7
util-linux 2.23.2-33.el7 Not Installed
usermode 1.111-5.el7 Not Installed
yum-metadata-parser 1.1.4-10.el7 Not Installed
pygobject3-base 3.14.0-3.el7 Not Installed
dracut 033-463.el7 Not Installed
rootfiles 8.1-11.el7 Not Installed
ustr 1.0.4-16.el7 Not Installed
elfutils-libs 0.166-2.el7 Not Installed
diffutils 3.3-4.el7 Not Installed
dbus 1.6.12-17.el7 Not Installed
libuuid 2.23.2-33.el7 Not Installed
gdb-gdbserver 7.6.1-94.el7 Not Installed
libmount 2.23.2-33.el7 Not Installed
libxml2-python 2.9.1-6.el7_2.3 Not Installed
yum 3.4.3-150.el7 Not Installed
virt-what 1.13-8.el7 Not Installed
libdnf Not Installed 0.7.4-2.el7.el
libsemanage 2.5-5.1.el7_3 Not Installed
gzip 1.5-8.el7 Not Installed
passwd 0.79-4.el7 Not Installed
python-kitchen 1.1.1-5.el7 Not Installed
libnl 1.1.4-3.el7 Not Installed
binutils 2.25.1-22.base.el7 Not Installed
acl 2.2.51-12.el7 Not Installed

Here you’ll see there are 80 package differences. Six packages have been added to support the new package manager: librhsm, json-glib, libsolv, microdnf, librepo and libdnf.74 packages have been removed leaving just the minimum set of packages.

Out of interest I wanted to see how this package list differed from Oracle’s slim image.

Package Oracle Linux Slim RHEL Atomic
python-chardet 2.2.1-1.el7_1 Not Installed
nss-tools 3.21.3-2.0.1.el7_3 3.21.3-2.el7_3
python-urlgrabber 3.10-8.el7 Not Installed
libxml2 2.9.1-6.0.1.el7_2.3 2.9.1-6.el7_2.3
audit-libs 2.6.5-3.el7 Not Installed
nss-sysinit 3.21.3-2.0.1.el7_3 3.21.3-2.el7_3
file-libs 5.11-33.el7 Not Installed
rpm-build-libs 4.11.3-21.el7 Not Installed
python-libs 2.7.5-48.0.1.el7 Not Installed
json-glib Not Installed 1.0.2-1.el7
gdbm 1.10-8.el7 Not Installed
nss 3.21.3-2.0.1.el7_3 3.21.3-2.el7_3
pyxattr 0.5.1-5.el7 Not Installed
yum-plugin-ovl 1.1.31-40.el7 Not Installed
basesystem 10.0-7.0.1.el7 10.0-7.el7
pygpgme 0.3-9.el7 Not Installed
coreutils 8.22-18.0.1.el7 8.22-18.el7
shadow-utils 4.1.5.1-24.el7 Not Installed
python-pycurl 7.19.0-19.el7 Not Installed
libcap-ng 0.7.5-4.el7 Not Installed
bash 4.2.46-21.0.1.el7_3 4.2.46-21.el7_3
python-iniparse 0.4-9.el7 Not Installed
microdnf Not Installed 2-3.el7.1.1
librhsm Not Installed 0.0.1-1.el7
kernel-container 3.10.0-0.0.0.2.el7 Not Installed
gobject-introspection Not Installed 1.42.0-1.el7
python 2.7.5-48.0.1.el7 Not Installed
cpio 2.11-24.el7 Not Installed
yum-utils 1.1.31-40.el7 Not Installed
pyliblzma 0.5.3-11.el7 Not Installed
rpm-python 4.11.3-21.el7 Not Installed
librepo Not Installed 1.7.16-1.el7
yum-metadata-parser 1.1.4-10.el7 Not Installed
libsolv Not Installed 0.6.20-5.el7
ustr 1.0.4-16.el7 Not Installed
oraclelinux-release 7.3-1.0.4.el7 Not Installed
diffutils 3.3-4.el7 Not Installed
redhat-release-server 7.3-7.0.1.el7 7.3-7.el7
libxml2-python 2.9.1-6.0.1.el7_2.3 Not Installed
yum 3.4.3-150.0.1.el7 Not Installed
libdnf Not Installed 0.7.4-2.el7.el
libsemanage 2.5-5.1.el7_3 Not Installed
python-kitchen 1.1.1-5.el7 Not Installed
gpg-pubkey ec551f03-53619141 Not Installed

Ignoring the version differences, there are 7 packages in RHEL Atomic not present in the Oracle Slim image which support the new microdnf package manager. There are 28 packages in Oracle Slim that are not in the RHEL Atomic image – unsurprisingly most of these relate to the inclusion of YUM. It will be interesting to see if Oracle Linux and the other RHEL derivatives follow suit and use microdnf in their images.

This is a great step forward for RHEL users, reducing the image size and the attack surface but still leaves a lot of, arguably, unnecessary content in the image. Take a look at the files view in the content tab of the Anchore Navigator for this image here and, as we did for Alpine earlier, filter for /bin to see the utilities and other libraries installed in the image.

At this point the challenge in reducing the image further is that most of the packages left are required either in whole, or more likely in part due to dependencies. For example you could argue that there is no good reason to have the /bin/chmod command in the image however that is part of the coreutils package which is required by multiple other packages so any further steps forward will require some major changes in packaging.

If you have a Red Hat Enterprise Linux subscription I’d encourage you to check out the new Atomic base image and see how you can reduce the footprint and attack surface of your RHEL based images.

And whether you use RHEL, CentOS, Debian, Ubuntu, Alpine or other distributions you can use Anchore’s image analysis and compliance tools to ensure that the images you deploy meet your security and best practices requirements.

There are several ways you can get started with Anchore. Click the button below to learn more and install our open source project, or if you’d like to try the free Anchore Image Navigator click here or go to https://anchore.io/.

Have a Question? Send us a Message.