July 2016

Signed, Sealed, Deployed

Posted on July 29, 2016December 20, 2021 by Christian Wiens

Red Hat recently blogged about their progress in adding support for container image signing, a particularly interesting and most welcome aspect of the design is the way that the binary signature file can be decoupled from the registry and distributed separately. The blog makes interesting reading and I’d strongly recommend reading through it, I’m sure you’ll appreciate the design. And of course, code is available online.

Red Hat is, along with the other Linux distributors, well versed in the practice of signing software components to allow end-users to verify that they are running authentic code and is in the process of extending this support to container images. The approach described is different from that taken previously by Docker Inc. however rather than comparing the two approaches I wanted to talk at a high level about the benefits of image signing along with some commentary about trust.

In the physical world, we are all used to using our signature to confirm our identity.

Probably the most common example is when we are signing a paper check or using an electronic signature pad during a sales transaction. How many times have you signed your signature so quickly that you do not even recognize it yourself? How many times in recent memory has a cashier or server compared the signature written with that on the back of your credit card? In my experience that check is likely to happen one out of every ten times and even in those cases that check is little more than a token gesture and the two signatures may not have matched.

That leads me to the first important observation: a signature mechanism is only useful if it is checked. Obviously, when vendors such as Docker Inc, Red Hat, and others implement an image signing and validation system, the enforcement will be built into to all layers, so that in one example a Red Hat delivered image will be validated by a Red Hat provided Docker runtime to ensure it’s signed by a valid source.

However it’s more likely that the images that you deploy in your enterprise won’t just be the images downloaded from a registry, but instead, images built on top of these images or perhaps even built from scratch, so for image signing to provide the required level of security all images created within your enterprise should also be signed and have those signatures validated before the image is deployed. Some early users of image signing that we have talked to have used image signing less as a way of tracking the provenance of images but instead as a method to show that an image has not been modified between leaving the CI/CD pipeline and being deployed on their container host.

Before we dig into the topic of image signing it’s worth discussing what a signature actually represents.

The most common example of signatures that we see in our day to day life is in our web browsers where we look for the little green padlock in the address bar that indicates that the connection to the webserver from our browser is encrypted but most importantly it confirms that you are talking to the expected website.

The use of TLS/SSL certificates allows your browser to validate that when you connect to https://www.example.com the content displayed actually came from example.com.

So in this example, the signature was used to confirm the source of the (web) content. Over many years we have been trained NOT to type our credit card details into a site that is NOT delivered through HTTPS.

But that does not mean that you would trust your credit card details to any site that uses HTTPS.

The same principle applies to the use of image signatures. If you download an image signed by Red Hat, Docker Inc, or any other vendor, you can be assured that the image did come from this vendor. The level of confidence you have in the contents of the image is based on the level of trust you already have with the vendor. For example, you are likely not to run an image signed by l33thackerz even though it may include a valid signature.

As enterprises move to a DevOps model with containers we’re seeing a new software supply chain, which often begins with a base image pulled from DockerHub or a vendor registry.

This base image may be modified by the operations team to include extra packages or to customize specific configuration files. The resulting image is then published in the local registry to be used by the development team as the base image for their application container. In many organizations, we are starting to see other participants in this supply chain, for example, a middleware team may publish an image containing an application server that is in turn used by an application team.

For the promise of image signing to be fulfilled, at each stage of this supply chain, each team must sign the image to ensure that the ‘chain of custody’ can be validated throughout the software development lifecycle. As we covered previously those signatures only serve to prove the source of an image, during any point in the supply chain from the original vendor of the base image all the way through the development process the images may be modified. At any step in the supply chain a mistake may be made, an outdated package that contains known bugs of vulnerabilities may be used, an insecure configuration option in an application’s configuration file or perhaps secrets such as passwords or API keys may be stored in the image.

Signing an image will not prevent insecure or otherwise non-compliant images from being deployed, however, as part of a post mortem, it will provide a way of tracking down when the vulnerability or bug was introduced.

During each stage of the supply chain, detailed checks should be performed on the image to ensure that the image complies with your site-specific policies.

These policies could cover security, starting with the ubiquitous CVE scan but then going further to analyze the configuration of key security components. For example, you could have the latest version of the apache web server but have configured the wrong set of TLS Ciphers suites leading to insecure communication. In addition to security, your policies could cover application-specific configurations to comply with best practices or to enable consistency and predictability.

Anchore’s goal is to provide a toolset that allows developers, operations, and security teams to maintain full visibility of the ‘chain of custody’ as containers move through the development lifecycle while providing the visibility, predictability, and control needed for production deployment.

With Anchore’s tools, the analysis and policy evaluation could be run during each stage of the supply chain allowing the signatures to attest to both the source of the image and also the compliance of the image’s contents.

In summary, we believe that image signing is an important part of the security and integrity of your software supply chain however signatures alone will not ensure the integrity of your systems.

Webinar – Introduction to the Anchore Project

Posted on July 21, 2016December 20, 2021 by Christian Wiens

Today we delivered Anchore’s first webinar where we gave an introduction into Anchore’s open source project and discussed how we can democratize certification through the use of open source.

A primary concern for enterprises adopting Docker is security most notably, in the governance and compliance of the containers that they are deploying. In the past, as we moved from physical server deployments to virtual machines we saw similar issues and we spoke about “VM sprawl” but containers are set to exponentially outgrow VM deployments. It’s almost too easy to pull an application image from a public registry and run it, within seconds you can deploy an application in production without even knowing what’s under the covers.

Organizations want to have confidence in their deployments, to know that when they deploy an application it will work, it will be secure, it can be maintained and it will be performant.

In the past, this confidence came through certification. Commercial Linux distributions such as Red Hat, SuSE and others set the standard and worked with hardware and software vendors on certification programs to give a level of assurance to end-users that the operating system would run reliably on their hardware and also offer insurance in the form of enterprise-grade commercial support if they encountered issues.

Today the problem is more complex and there can no longer be just a single certification. For example, the requirements of a financial services company are different from the requirements of a healthcare company handling medical records and these are different from the needs of a federal institution and so on. Even the needs of individual departments within any given organization may be different.

What is needed now is the ability for IT operations and security to be able to define their own certification requirements which may differ even from application to application, allowing them to define these policies and evaluate them before applications are deployed into production.

What we are talking about is the democratization of certification

Rather than having certification in the hands of a small number of vendors or standards bodies, we want to allow organizations to define what certification means to them.

Please tune into the webinar where we go a level deeper to discuss the challenges around container certification, how an open source, democratized approach can help end-users and introduce our open source tooling.

Extending Anchore with Lynis

Posted on July 19, 2016December 20, 2021 by Daniel Nurmi

Add Lynis Scanning to Anchore Image Analysis

Note: You will need the latest Anchore code from GitHub to follow this procedure: Install it here

In this post, we focus on solving a common problem that is faced when building out a container-based deployment environment – take an existing tool/practice for deciding whether or not application code is ready to be deployed, and apply it to the steady stream of container images that are flowing in from developers on their way to production. With Anchore, we show that we can apply many existing tools/techniques to container images easily, in a way that leads to a ‘fail fast’ property where things can be checked early on in the CI/CD pipeline (pre-execution).

To illustrate this idea, we walk through the the process of adding a new analyzer/gate to Anchore – specifically I would like to include the scanning of all container images using the ‘Lynis’ open-source Linux Distro scanning utility, and then be able to use the Anchore policy system to make decisions based on the result of the Lynis scan. Once complete, every container image that is analyzed by Anchore in the future will include a lynis report, and every analyzed image will be subject to the Lynis gate checker.

The process is broken down into two parts – first, we write an ‘analyzer’ that is responsible for running the Lynis scan whenever any container is analyzed with Anchore, and second we write a ‘gate’ which can take as input the result of the Lynis scan and emits triggers based on what it finds. From there, we can then use the normal Anchore policy strings to make STOP/WARN/GO suggestions based on what triggers the gate emits.

Writing the Lynis Analyzer Module

First, I use the anchore tool to set up a module development environment.

Note the output where it shows the exact paths on your system.  I run the exact command just to make sure everything is sane:

# /tmp/3355618.anchoretmp/anchore-modules/analyzers/analyzer-example.sh 0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618 /tmp/3355618.anchoretmp/data /tmp/3355618.anchoretmp/data/0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618 /tmp/3355618.anchoretmp

RESULT: pfiles found in image, review key/val data stored in:

/tmp/3355618.anchoretmp/data/0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618/analyzer_output/analyzer-example/pfiles

Since I want to write a python module (instead of the included example shell script), I’ll start with an existing anchore python analyzer script and call it ’10_lynis_report.py’

# cp /usr/lib/python2.7/site-packages/anchore/anchore-modules/analyzers/10_package_list.py /tmp/3355618.anchoretmp/anchore-modules/analyzers/10_lynis_report.py

I’ll trim most of the code out, and change the ‘analyzer_name’ to a new name for this module – I’ve chosen ‘lynis_report’.

Next, I’ll add my code which first downloads the lynis scanner from a URL and creates a tarball that contains lynis. Then, the code uses an anchore utility routine that takes an input tarball and the input container image, and runs an instance of the container with the input tarball stages and available, executing the lynis scanner. Finally, the routine returns the stdout/stderr output of the executed container along with the contents of a specified file from within the container (in this case, the lynis report data itself). The last thing the analyzer does is write the lynis report data to the anchore output directory for later use.

While writing this code, we use the follow command each time to iterate and get the analyzer working the way we would like (i.e. when the lynis.report output file contains the lynis report data itself, we know the analyzer is working properly)

# /tmp/3355618.anchoretmp/anchore-modules/analyzers/10_lynis_report.py 0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618 /tmp/3355618.anchoretmp/data /tmp/3355618.anchoretmp/data/0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618 /tmp/3355618.anchoretmp

# cat /tmp/3355618.anchoretmp/data/0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618/analyzer_output/lynis_report/lynis.report

The finished module is here:

#!/usr/bin/env python

import sys
import os
import shutil
import re
import json
import time
import rpm
import subprocess
import requests
import tarfile

import anchore.anchore_utils

analyzer_name = “lynis_report”

try:
config = anchore.anchore_utils.init_analyzer_cmdline(sys.argv, analyzer_name)
except Exception as err:
print str(err)
sys.exit(1)

imgname = config
outputdir = config
unpackdir = config

if not os.path.exists(outputdir):
os.makedirs(outputdir)

try:
#datafile_dir = ‘/’.join(, ‘datafiles’])
datafile_dir = ‘/tmp/’
url = ‘https://cisofy.com/files/lynis-2.2.0.tar.gz’
r = requests.get(url)
TFH=open(‘/’.join(), ‘w’);
TFH.write(r.content)
TFH.close()

lynis_data_tarfile = ‘/’.join()
tar = tarfile.open(lynis_data_tarfile, mode=’w’, format=tarfile.PAX_FORMAT)
tar.add(‘/’.join(), arcname=’/lynis.tgz’)
tar.close()

except Exception as err:
print “ERROR: cannot locate datafile directory for lynis staging: ” + str(err)
sys.exit(1)

FH=open(outputdir + “/lynis.report”, ‘w’)
try:
fileput = lynis_data_tarfile
(o, f) = anchore.anchore_utils.run_command_in_container(image=imgname, cmd=”tar zxvf /lynis.tgz && cd /lynis && sh lynis audit system –quick”, fileget=”/var/log/lynis-report.dat”, fileput=fileput)
FH.write(‘ ‘.join([“LYNIS-REPORT-JSON”, json.dumps(f)]))
except Exception as err:
print str(err)

FH.close()

NOTE: this module is the basic code only meant as a demonstration, it does not include any checking for errors/faults as this would add a bit of code unrelated to the purpose of this posting.

Writing the Lynis Gate Module

The process of writing a gate is very similar to writing an analyzer – there are a few input differences and output file expectations, but the general process is the same. I will start with an existing anchore gate modules and trim the functional code:

# cp /usr/lib/python2.7/site-packages/anchore/anchore-modules/gates/20_check_pkgs.py /tmp/3355618.anchoretmp/anchore-modules/gates/10_lynis_gate.py

Here is the module with the functional code trimmed out:

#!/usr/bin/env python

import sys
import os
import re
import anchore.anchore_utils

try:
config = anchore.anchore_utils.init_gate_cmdline(sys.argv, “LYNIS report checker”)
except Exception as err:
print str(err)
sys.exit(1)

if not config:
sys.exit(0)

imgid = config
imgdir = config
analyzerdir = config
comparedir = config
outputdir = config

try:
params = config
except:
params = None

if not os.path.exists(imgdir):
sys.exit(0)

# code will go here

sys.exit(0)

Next, we need to set up the input by putting the imageId that we’re testing against into an input file for the gate, and then we can run the module manually and check the output iteratively until we’re happy.

# echo 0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618 > /tmp/3355618.anchoretmp/querytmp/inputimages

# /tmp/3355618.anchoretmp/anchore-modules/gates/10_lynis_gate.py /tmp/3355618.anchoretmp/querytmp/inputimages /tmp/3355618.anchoretmp/data/ /tmp/3355618.anchoretmp/data/0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618/gates_output/ PARAM=True

# cat /tmp/3355618.anchoretmp/data/0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618/gates_output/LYNISCHECK

The finished module is here:

#!/usr/bin/env python

import sys
import os
import re
import json
import traceback

import anchore.anchore_utils

try:
config = anchore.anchore_utils.init_gate_cmdline(sys.argv, “LYNIS report checker”)
except Exception as err:
traceback.print_exc()
print “ERROR: ” + str(err)
sys.exit(1)

if not config:
sys.exit(0)

imgid = config
imgdir = config
analyzerdir = config
comparedir = config
outputdir = config

try:
params = config
except:
params = None

if not os.path.exists(imgdir):
sys.exit(0)

# code will go here

output = ‘/’.join()
OFH=open(output, ‘w’)

try:
FH=open(‘/’.join(), ‘r’)
lynis_report = False
for l in FH.readlines():
l = l.strip()
(k, v) = re.match(‘(S*)s*(.*)’, l).group(1, 2)
lynis_report = json.loads(v)
FH.close()

if lynis_report:
for l in lynis_report.splitlines():
l = l.strip()
if l and not re.match(“^s*#.*”, l) and re.match(“.*=.*”, l):
(k, v) = re.match(‘(S*)=(.*)’, l).group(1, 2)
if str(k) == ‘warning[]’:
# output a trigger
OFH.write(‘LYNISWARN ‘ + str(v) + ‘n’)
elif str(k) == ‘suggestion[]’:
OFH.write(‘LYNISSUGGEST ‘ + str(v) + ‘n’)
elif str(k) == ‘vulnerable_package[]’:
OFH.write(‘LYNISPKGVULN ‘ + str(v) + ‘n’)

except Exception as err:
traceback.print_exc()
print “ERROR: ” + str(err)

OFH.close()
sys.exit(0)

NOTE: this module is the basic code only meant as a demonstration, it does not include any checking for errors/faults as this would add a bit of code unrelated to the purpose of this posting.

Tie the Two Together

Now that we’re finished writing and testing the module, we can drop the new analyzer/gate modules into anchore and use the anchore CLI as normal. First we copy the new modules into a location where anchore can use them:

cp /tmp/3355618.anchoretmp/anchore-modules/analyzers/10_lynis_report.py ~/.anchore/user-scripts/analyzers/
cp /tmp/3355618.anchoretmp/anchore-modules/gates/10_lynis_gate.py ~/.anchore/user-scripts/gates/

Next, we run the normal analyze operation which will now include the lynis analyzer:

anchore analyze –force –image ubuntu –imagetype none

Then, we can add new lines to the image’s policy that describe what actions to output if the new gate emits its triggers:

anchore gate –image ubuntu –editpolicy

# opens an editor, where you can add the following lines to the existing image’s policy
LYNISCHECK:LYNISPKGVULN:STOP
LYNISCHECK:LYNISWARN:WARN
LYNISCHECK:LYNISSUGGEST:GO

Finally, we can run the normal anchore gate, and see the resulting triggers showing up alongside the other anchore gates:

anchore gate –image ubuntu

Peek Into Your Containers With 3 Simple Commands

Posted on July 13, 2016April 13, 2022 by Christian Wiens

If you are just looking to run a common Linux application such as Tomcat or WordPress it’s far simpler to download a pre-packaged image from DockerHub than to install the application from scratch. But with tens of thousands of images on DockerHub you are likely to find many variations of the application in question, even with official repositories you may find multiple different versions of an application.

In previous blog posts, we have introduced the Anchore open source project which provides a rich toolset to allow developers, operations, and security teams to maintain full visibility of the ‘chain of custody’ as containers move through the development lifecycle.

In our last blog, we covered a couple of simple use cases, allowing a user to dig into the contents of a container looking at specific files or packages. In this blog post, I wanted to introduce you to three interesting features within Anchore.

There are seven top-level commands within the Anchore command-line tools. These can be seen by running the anchore command with no other options:

Command	Description
`analyze`	Perform analysis on specified image IDs
`explore`	Search, report and query specified image IDs
`gate`	Perform and view gate evaluation on selected images
`subscriptions`	Manage local subscriptions
`sync`	Synchronize images and metadata
`system`	Anchore system-level operations
`toolbox`	Useful tools and operations on images and containers

In previous blog posts, we have presented the analyze, explore and gate commands, but in this blog post, we wanted to highlight a couple of the lesser-known features in the toolbox that we found very useful in our day to day use of containers.

Running anchore toolbox will show the sub-commands available:

Command	Description
`setup-module-dev`	Setup a module development environment
`show`	Show image summary information
`show-dockerfile`	Generate (or display actual) image Dockerfile
`show-familytre`	Show image family tree image IDs
`show-layers`	Show image layer IDs
`show-taghistory`	Show history of all known repo/tags for image
`unpack`	Unpack and Squash image to local filesystem

While Docker allows applications to be packaged as easily distributed containers, transparently providing both the underlying operating system and the application, you often need to know exactly what operating system this application is built upon. This information may be required to fulfill compliance or audit requirements in your organization or to ensure that you are only deploying operating systems for which you have commercial support agreements.

If you are lucky then the full description of the container on the DockerHub portal contains details about the operating system used. But in many cases, this information isn’t presented.

One way to ascertain what operating system is used is to download and run the image and inspect the file system however that’s a manual and time-consuming process. The show command presents a simple way to retrieve this information.

Taking a look at nginx, the most popular image on DockerHub:

IMAGEID='0d409d33b27e47423b049f7f863faa08655a8c901749c2b25b93ca67d01a470d'
REPOTAGS='docker.io/nginx:latest'
DISTRO='debian'
DISTROVERS='8'
SHORTID='0d409d33b27e'
PARENTID=''
BASEID='0d409d33b27e47423b049f7f863faa08655a8c901749c2b25b93ca67d01a470d'

Here we see the latest image is built on Debian version 8 (Jessie).

Using another useful toolbox function is show-tag history, which shows the known tags for a given image. Here we can see that the latest image is also tagged as 1.11 and 1.11.1:

+--------------+---------------------+-------------------------+
|   ImageId    |         Date        |        KnownTags        |
+--------------+---------------------+-------------------------+
| 0d409d33b27e | Wed Jun 15 14:37:03 | nginx:1.11,nginx:1.11.1 |
|              |         2016        |      ,nginx:latest      |
| 0d409d33b27e | Wed Jul 13 15:57:51 | nginx:1.11,nginx:1.11.1 |
|              |         2016        |      ,nginx:latest      |
| 0d409d33b27e | Wed Jul 13 16:35:14 |  docker.io/nginx:latest |
|              |         2016        |                         |
+--------------+---------------------+-------------------------+

The final toolbox feature I want to highlight is a feature that many users do not know is available, that is the ability to retrieve the dockerfile for a given image. The show-dockerfile command will either display the dockerfile, if it was available during the image analysis phase, or will generate the dockerfile from the image.

This information may be useful if you wish to look under the covers to understand how the container was created or to check for any potential issues with the container content. The contents of the dockerfile may also be used within our ‘gates’ feature, for example allowing you to specify that specific ports may not be exposed.

# anchore toolbox --image=nginx:latest show-dockerfile
--- ImageId ---
0d409d33b27e

--- Mode ---
Guessed

Here the mode Guessed indicates that the dockerfile was generated by the tool during image analysis.

There are other toolbox commands that include the ability to show the family tree of an image, display the image layers, or unpack the image to the local filesystem.

If you haven’t already installed Anchore and begun scanning your container images, take a look at our installation and quick-start guides at our wiki below or by going to https://github.com/anchore/anchore/wiki.