June 2018

Container Security & Compliance Scanning For Codeship

Posted on June 19, 2018January 5, 2022 by Jeremy Valance

This will walk through integrating Anchore scanning into a Codeship pipeline. During the first step, a Docker image will be built from a Dockerfile. Following this, during the second step Anchore will scan the image, and depending on the result of the policy evaluation, proceed to the final step. During the final step, the built image will be pushed to a Docker registry.

Prerequisites

Docker Installed
Jet CLI
Codeship Pro Account
Github Account
Anchore Engine

Setup

Prior to setting up your Codeship build pipeline, an Anchore Engine service needs to be accessible from the pipeline. Typically this is on port 8228. In this example, I have an Anchore Engine service on AWS EC2 with standard configuration. I also have a Dockerfile in a Github repository that I will build an image from during the first step of the pipeline. In the final step, I will be pushing the built image to an image repository in my personal Dockerhub.

The Github repository can be referenced here.

Repository contents:

codeship-services.yml (Contains all services needed to run your CI/CD builds)
codeship-steps.yml (Contains all the steps for your CI/CD process)
Dockerfile
dockercfg.encrypted (Docker registry credentials)
env.encrypted (Environment variables)

For more info on using encrypted files with Jet CLI visit here.

Most typically, we advise on having a staging registry and production registry. Meaning, being able to push and pull images freely from the staging/dev registry while maintaining more control over images being pushed to the production registry. In this example, I am using the same registry for both.

I’ve added the following environment variables via the envfile:

If ANCHORE_FAIL_ON_POLICY is set to true, the pipeline will fail, and the image will not be pushed to the registry.

ANCHORE_CLI_URL
ANCHORE_CLI_USER
ANCHORE_CLI_PASS
ANCHORE_CLI_IMAGE
ANCHORE_RETRIES
ANCHORE_FAIL_ON_POLICY

The Docker registry has been configured with the dockercfg file:

{
	"auths": {
		"https://index.docker.io/v1/": {
			"auth": "anZhbGFuY2U6MjI2MTM3QGtLaw=="
		}
	},
	"HttpHeaders": {
		"User-Agent": "Docker-Client/17.10.0-ce (linux)"
	}
}

Build Image

In the first step of the pipeline, we build a Docker image from a Dockerfile as defined in our codeship-steps.yml:

- name: imagebuildstep
  service: imagebuild
  type: push
  image_name: jvalance/sampledockerfiles
  encrypted_dockercfg_path: dockercfg.encrypted

and our codeship-services.yml:

imagebuild:
  build:
    dockerfile: Dockerfile
  cached: true

Conduct Anchore Scan

In the second step of the pipeline, we scan the built image with Anchore as defined in our codeship-steps.yml:

- name: anchorestep
  service: anchorescan
  command: sh -c 'echo "Adding image to Anchore engine" && 
    anchore-cli image add $ANCHORE_IMAGE_SCAN &&
    echo "Waiting for image analysis to complete" &&
    counter=0 && while (! (anchore-cli image get $ANCHORE_IMAGE_SCAN | grep 'Status: analyzed') ) ; do echo -n "." ; sleep 10 ; if  ; then echo " Timeout waiting for analysis" ; exit 1 ; fi ; counter=$(($counter+1)) ; done &&
    echo "Analysis complete" &&
    if  ; then anchore-cli evaluate check $ANCHORE_IMAGE_SCAN  ; fi'
  encrypted_env_file: env.encrypted

and our codeship-services.yml:

anchorescan:
  image: anchore/engine-cli:latest
  encrypted_env_file: env.encrypted

Depending on the output of the policy evaluation, the pipeline may or may not fail. In this case, I have set ANCHORE_FAIL_ON_POLICY to true and exposed port 22. This is in violation of a policy rule, so the build will fail during this step.

Push Image

In the final step of the pipeline, we push the Docker image to a registry as defined in the codeship-steps.yml:

- name: imagepushstep
  service: imagebuild
  type: push
  image_name: jvalance/sampledockerfiles
  encrypted_dockercfg_path: dockercfg.encrypted

and our codeship-services.yml:

anchorescan:
  image: anchore/engine-cli:latest
  encrypted_env_file: env.encrypted

As a reminder, we advise having separate Docker registries for images that are being scanned with Anchore, and images that have passed an Anchore scan. For example, a registry for dev/test images, and a registry to certified, trusted, production-ready images.

Anchore & Falco, End-to-End OSS Container Security Solution

Posted on June 12, 2018December 19, 2021 by Team Anchore

While open source solutions have historically provided the core layer of infrastructure, there have been areas in which organizations would need to look at proprietary solutions. The most notable of which is a security that had until recently remained the bastion of commercial vendors.

For container infrastructure there are typically two key security needs:

1. Image Security

Analyzing images to ensure they do not contain vulnerabilities and are in compliance with your organization’s operational and security policies.

2. Runtime Security

Real-time monitoring of containers to ensure report on or block malicious activity at the network, system or storage layers.

We have spoken at length about the first area: image security and covered how the open source Anchore Engine can quickly and easily be integrated into your CI/CD pipeline, container registries and Kubernetes infrastructure to ensure that only images that meet your organization’s policies are deployed. In this blog, we will introduce you to another open source project, Falco, from the team at Sysdig. Like Anchore Engine, Falco is open source, making it easy for organizations to download and run Falco in their environment and like Anchore there is company behind Falco that provides a commercial offering with centralized management, added features and integration.

Over the last decade, it has become clear that open source technologies provide the right foundation for infrastructure and at Anchore we believe that security and analysis tools should be open source so that anyone can inspect the code to validate that the results are fair and accurate. And since security tools typically are granted the highest level of privilege in terms of access and control of resources you need the mantra of “Trust but verify” is especially true.

With Anchore Engine, we ensure that only the right content, from known sources configured in the right way, is promoted from your CI/CD system and deployed in production but once deployed unknown vulnerabilities or misconfigurations can lead to a container being exploited. The traditional approach to security monitoring involved looking for known signatures is network traffic, files, etc. Similar to the approach taken in the early days of antivirus software where security vendors played an endless game of cat and mouse with virus authors, requiring the antivirus software to be continually updated with new signatures and new viruses were detected in the wild. Over time these solutions evolved to use heuristics in addition to signature mapping.

A similar technique is used by the Falco project which takes a more behavioral approach to detection. While there are many different ways that a container could be compromised all of which would need to be explicitly monitored for Falco looks at what is happening once the attacker has compromised the container allowing you to report and then block anomalous behaviors. For example, why would a reverse proxy container need to write a file into the /bin directory, why would a PostgreSQL container make an outbound network connection, why would your Redis server spawn a shell process?

Falco taps into host kernel for syscall monitoring using either a kernel module or a new approach using extended Berkley Packaged Filters (eBPF) which is available in modern kernels (see an excellent introduction to eBFG in LWN). This approach maximizes visibility into the system while minimizing overhead. Rules can be created that can monitor any activity including network access, file I/O and even interprocess communication (IPC). The Falco Wiki contains some great examples that illustrate the power of this level of integration, for example alerting when a process attempts to write into a directory containing system binaries, they even created default runtime security rules for the most popular Docker images.

With the addition of Anchore Engine and Sysdig Falco you can build an open and secure container infrastructure.

How Often are Docker Images Updated – Revisited

Posted on June 8, 2018January 5, 2022 by Alex Springer

Refreshing the Data

Almost a year ago we looked at the frequency with which some of the most popular images in the Docker registry are updated and compared the frequency of base image (alpine, debian, etc) updates with that of popular non-OS images. We found that while updates to base images came in around once a month, non-OS images updated much more frequently – up to eight to ten updates in the case of the widely used node:latest and php:latest packages.

We learned that while all official images should follow DockerHub best practices and should, therefore, be well maintained it is clear from our historic data that many images can be updated infrequently and carry security vulnerability for many weeks. Understanding these gaps in the update is crucial to a comprehensive and container and application security policy, so we decided it was time to take a second look.

Using Anchore Cloud (anchore.io) makes it easy to check the update history of each image:

This lets us easily see the image creation date as well as the date of the most recent Anchore analysis.

The Anchore service continually polls DockerHub and when an update to a repository has detected the list of tags and images are retrieved and any new images are downloaded and analyzed. In addition, Anchore scans images on a regular cadence to ensure that the results of each scan include the latest CVEs and other known vulnerabilities.

We went ahead and pulled the data from the last 18 months for the library/debian:latest images from the database:

Our last analysis was in September of 2017 you can see that since then updates have become more infrequent. We observed in September that while a monthly cadence can seem ideal, what really matters is the timeliness of updates, particularly updates that fix new critical vulnerabilities. With updates coming less frequently (as seen above) this image becomes less likely to have rapidly addressed any issues that may have arisen.

Users of these base images need to be proactive about their own image fixes during these periods to avoid exposure. Tools like Anchore.io allow users of images to subscribe to the results of each Anchore CVE analysis, an analysis that is conducted regularly in-between image updates and well as immediately after updates occur.

Update Frequency – Most Popular Images

Let’s look at the relative update frequency across popular base images:

We see a similar pattern across the major operating system repositories. None have a fixed update schedule, and while some such as Ubuntu and Oracle Linux are consistent, repositories like Fedora and Alpine can go up to four months without an update!

As we pointed out in our previous post on this topic, these gaps do not immediately imply vulnerabilities. Lightweight operating systems like Alpine and BusyBox require less maintenance due to the relatively small number of packages and therefore potential vulnerabilities. However, if an image hasn’t been updated it is always worth analyzing and confirming the image you are using for production applications is still protected.

An encouraging pattern to look for is present in the Ubuntu and Oracle Linux update timings as well. In addition to the semi-regular monthly updates, updates are pushed at seemingly random times throughout the month. These updates each represent opportunities for the various teams to address new security concerns in a timely fashion. Anchore’s regular post-update scans serve to confirm this to be the case.

Moving on to some popular non-OS images we see a much greater update frequency. The increased complexity in the images necessitates these shortened update cycles and ideally, there is also an update that follows soon after the underlying base image is updated. This is not always the case our analysis shows that some applications are not rebuilt for several weeks or are rebuilt on top of an older base image.

Our assessment from September holds true today. While all official images should follow DockerHub best practices and be well maintained many images can be updated infrequently and carry security vulnerabilities for many weeks.

The timing of an image update can be an indicator of the health of the image, but the content of an image is even more important. Check out the Anchore Cloud to explore image contents, update timelines, and vulnerabilities as well as subscribe to the analysis of the images you use every day.

If you’d like to do this scanning on-prem, check out the open source anchore-engine.

For a great follow-up and to help understand the best course of action to take when it comes to determining how often you update the images your applications rely on, check out Just because they pushed doesn’t mean you need to pull.