Extending Anchore with Lynis

Add Lynis Scanning to Anchore Image Analysis

Note: You will need the latest Anchore code from GitHub to follow this procedure: Install it here

In this post, we focus on solving a common problem that is faced when building out a container-based deployment environment - take an existing tool/practice for deciding whether or not application code is ready to be deployed, and apply it to the steady stream of container images that are flowing in from developers on their way to production. With Anchore, we show that we can apply many existing tools/techniques to container images easily, in a way that leads to a ‘fail fast’ property where things can be checked early on in the CI/CD pipeline (pre-execution).

To illustrate this idea, we walk through the the process of adding a new analyzer/gate to Anchore - specifically I would like to include the scanning of all container images using the 'Lynis' open-source Linux Distro scanning utility, and then be able to use the Anchore policy system to make decisions based on the result of the Lynis scan. Once complete, every container image that is analyzed by Anchore in the future will include a lynis report, and every analyzed image will be subject to the Lynis gate checker.

The process is broken down into two parts - first, we write an 'analyzer' that is responsible for running the Lynis scan whenever any container is analyzed with Anchore, and second we write a 'gate' which can take as input the result of the Lynis scan and emits triggers based on what it finds. From there, we can then use the normal Anchore policy strings to make STOP/WARN/GO suggestions based on what triggers the gate emits.

Writing the Lynis Analyzer Module

First, I use the anchore tool to set up a module development environment.

Note the output where it shows the exact paths on your system.  I run the exact command just to make sure everything is sane:

# /tmp/3355618.anchoretmp/anchore-modules/analyzers/analyzer-example.sh 0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618 /tmp/3355618.anchoretmp/data /tmp/3355618.anchoretmp/data/0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618 /tmp/3355618.anchoretmp

RESULT: pfiles found in image, review key/val data stored in:

/tmp/3355618.anchoretmp/data/0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618/analyzer_output/analyzer-example/pfiles

Since I want to write a python module (instead of the included example shell script), I'll start with an existing anchore python analyzer script and call it '10_lynis_report.py'

# cp /usr/lib/python2.7/site-packages/anchore/anchore-modules/analyzers/10_package_list.py /tmp/3355618.anchoretmp/anchore-modules/analyzers/10_lynis_report.py

I'll trim most of the code out, and change the 'analyzer_name' to a new name for this module - I've chosen 'lynis_report'.

Next, I'll add my code which first downloads the lynis scanner from a URL and creates a tarball that contains lynis. Then, the code uses an anchore utility routine that takes an input tarball and the input container image, and runs an instance of the container with the input tarball stages and available, executing the lynis scanner. Finally, the routine returns the stdout/stderr output of the executed container along with the contents of a specified file from within the container (in this case, the lynis report data itself). The last thing the analyzer does is write the lynis report data to the anchore output directory for later use.

While writing this code, we use the follow command each time to iterate and get the analyzer working the way we would like (i.e. when the lynis.report output file contains the lynis report data itself, we know the analyzer is working properly)

# /tmp/3355618.anchoretmp/anchore-modules/analyzers/10_lynis_report.py 0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618 /tmp/3355618.anchoretmp/data /tmp/3355618.anchoretmp/data/0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618 /tmp/3355618.anchoretmp

# cat /tmp/3355618.anchoretmp/data/0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618/analyzer_output/lynis_report/lynis.report

The finished module is here:

#!/usr/bin/env python

import sys
import os
import shutil
import re
import json
import time
import rpm
import subprocess
import requests
import tarfile

import anchore.anchore_utils

analyzer_name = "lynis_report"

try:
config = anchore.anchore_utils.init_analyzer_cmdline(sys.argv, analyzer_name)
except Exception as err:
print str(err)
sys.exit(1)

imgname = config
outputdir = config
unpackdir = config

if not os.path.exists(outputdir):
os.makedirs(outputdir)

try:
#datafile_dir = '/'.join(, 'datafiles'])
datafile_dir = '/tmp/'
url = 'https://cisofy.com/files/lynis-2.2.0.tar.gz'
r = requests.get(url)
TFH=open('/'.join(), 'w');
TFH.write(r.content)
TFH.close()

lynis_data_tarfile = '/'.join()
tar = tarfile.open(lynis_data_tarfile, mode='w', format=tarfile.PAX_FORMAT)
tar.add('/'.join(), arcname='/lynis.tgz')
tar.close()

except Exception as err:
print "ERROR: cannot locate datafile directory for lynis staging: " + str(err)
sys.exit(1)

FH=open(outputdir + "/lynis.report", 'w')
try:
fileput = lynis_data_tarfile
(o, f) = anchore.anchore_utils.run_command_in_container(image=imgname, cmd="tar zxvf /lynis.tgz && cd /lynis && sh lynis audit system --quick", fileget="/var/log/lynis-report.dat", fileput=fileput)
FH.write(' '.join(["LYNIS-REPORT-JSON", json.dumps(f)]))
except Exception as err:
print str(err)

FH.close()

NOTE: this module is the basic code only meant as a demonstration, it does not include any checking for errors/faults as this would add a bit of code unrelated to the purpose of this posting.

Writing the Lynis Gate Module

The process of writing a gate is very similar to writing an analyzer - there are a few input differences and output file expectations, but the general process is the same. I will start with an existing anchore gate modules and trim the functional code:

# cp /usr/lib/python2.7/site-packages/anchore/anchore-modules/gates/20_check_pkgs.py /tmp/3355618.anchoretmp/anchore-modules/gates/10_lynis_gate.py

Here is the module with the functional code trimmed out:

#!/usr/bin/env python

import sys
import os
import re
import anchore.anchore_utils

try:
config = anchore.anchore_utils.init_gate_cmdline(sys.argv, "LYNIS report checker")
except Exception as err:
print str(err)
sys.exit(1)

if not config:
sys.exit(0)

imgid = config
imgdir = config
analyzerdir = config
comparedir = config
outputdir = config

try:
params = config
except:
params = None

if not os.path.exists(imgdir):
sys.exit(0)

# code will go here

sys.exit(0)

Next, we need to set up the input by putting the imageId that we're testing against into an input file for the gate, and then we can run the module manually and check the output iteratively until we're happy.

# echo 0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618 > /tmp/3355618.anchoretmp/querytmp/inputimages

# /tmp/3355618.anchoretmp/anchore-modules/gates/10_lynis_gate.py /tmp/3355618.anchoretmp/querytmp/inputimages /tmp/3355618.anchoretmp/data/ /tmp/3355618.anchoretmp/data/0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618/gates_output/ PARAM=True

# cat /tmp/3355618.anchoretmp/data/0f192147631d72486538039c51ef9557be11865030be2951a0fbe94ef66db618/gates_output/LYNISCHECK

The finished module is here:

#!/usr/bin/env python

import sys
import os
import re
import json
import traceback

import anchore.anchore_utils

try:
config = anchore.anchore_utils.init_gate_cmdline(sys.argv, "LYNIS report checker")
except Exception as err:
traceback.print_exc()
print "ERROR: " + str(err)
sys.exit(1)

if not config:
sys.exit(0)

imgid = config
imgdir = config
analyzerdir = config
comparedir = config
outputdir = config

try:
params = config
except:
params = None

if not os.path.exists(imgdir):
sys.exit(0)

# code will go here

output = '/'.join()
OFH=open(output, 'w')

try:
FH=open('/'.join(), 'r')
lynis_report = False
for l in FH.readlines():
l = l.strip()
(k, v) = re.match('(S*)s*(.*)', l).group(1, 2)
lynis_report = json.loads(v)
FH.close()

if lynis_report:
for l in lynis_report.splitlines():
l = l.strip()
if l and not re.match("^s*#.*", l) and re.match(".*=.*", l):
(k, v) = re.match('(S*)=(.*)', l).group(1, 2)
if str(k) == 'warning[]':
# output a trigger
OFH.write('LYNISWARN ' + str(v) + 'n')
elif str(k) == 'suggestion[]':
OFH.write('LYNISSUGGEST ' + str(v) + 'n')
elif str(k) == 'vulnerable_package[]':
OFH.write('LYNISPKGVULN ' + str(v) + 'n')

except Exception as err:
traceback.print_exc()
print "ERROR: " + str(err)

OFH.close()
sys.exit(0)

NOTE: this module is the basic code only meant as a demonstration, it does not include any checking for errors/faults as this would add a bit of code unrelated to the purpose of this posting.

Tie the Two Together

Now that we're finished writing and testing the module, we can drop the new analyzer/gate modules into anchore and use the anchore CLI as normal. First we copy the new modules into a location where anchore can use them:

cp /tmp/3355618.anchoretmp/anchore-modules/analyzers/10_lynis_report.py ~/.anchore/user-scripts/analyzers/
cp /tmp/3355618.anchoretmp/anchore-modules/gates/10_lynis_gate.py ~/.anchore/user-scripts/gates/

Next, we run the normal analyze operation which will now include the lynis analyzer:

anchore analyze --force --image ubuntu --imagetype none

Then, we can add new lines to the image's policy that describe what actions to output if the new gate emits its triggers:

anchore gate --image ubuntu --editpolicy

# opens an editor, where you can add the following lines to the existing image's policy
LYNISCHECK:LYNISPKGVULN:STOP
LYNISCHECK:LYNISWARN:WARN
LYNISCHECK:LYNISSUGGEST:GO

Finally, we can run the normal anchore gate, and see the resulting triggers showing up alongside the other anchore gates:

anchore gate --image ubuntu

Extending Anchore with Lynis

Add Lynis Scanning to Anchore Image Analysis

Writing the Lynis Analyzer Module

Writing the Lynis Gate Module

Tie the Two Together

From Chaos to Compliance: Revolutionizing License Management with Automation

An Outline for Getting Up to Speed on the DoD Software Factory

4 Ways to Prepare your Containers for the STIG Process

Speak with our security experts