Many users have already implemented Anchore to secure their CI/CD pipeline, to ensure that only images that are compliant with their security policies are pushed to their production registries. While this is a crucial process to implement on the path to implementing strong governance of container environments this only the first step.

It is not enough to just trust that the images in your production registry are compliant since while they may have passed policy checks at build time, images may fall out of compliance over time due to changes in policy or due to new vulnerabilities being discovered after the images were built and scanned. While many users rely on image signing to prove the provenance of images this does not guarantee that an image that was compliant when signed still complies with your security policies, while signing ensures that the image has not been changed policies and vulnerability data may change.

The next step in securing your container pipeline is to implement a system that ensures that only images that comply with your policies are deployed on your Kubernetes cluster or other orchestration platforms.

One approach to address this issue that we have seen is to provide a plugin for the Docker Daemon that is configured to communicate with a backend policy system. Command issues to the Docker Daemon will be filtered through this system and in the case of a failed image policy, the command to launch the container will be rejected. There are a number of challenges with this approach. Firstly this requires installing, configuring and maintaining plugins on every node, while this is certainly possible for Kubernetes systems where you have access and control of the underlying nodes, it becomes more complicated for hosted Kubernetes platforms where all that is exposed to the end-user is the API, such as Amazon’s upcoming EKS platform. With the growing list of OCI compatible runtimes, available plugins will have to be built for each runtime.

The main challenge with this plugin approach is that it takes control out of the hands of the orchestration platform rather than cooperating with the orchestrator.

Imagine the following scenario:

  • Your application is built on a Debian base image and deployed into production
  • A new high severity vulnerability is found in openssl is reported upstream.
  • A CVE is created and filed against the openssl package which is present in your image.
  • At this point, the image you use contains a vulnerability for which a fix may not be currently available
  • Your deployment needs to scale to add new replicas to handle load or perhaps compensate for a failure
  • Kubernetes tries to launch a container in a new pod however the container launch is blocked by the plugin.

In this example, you may wish to allow scale-out of new replicas of your pod but block creation of new deployments with resources that are out of compliance, otherwise, an unfixed CVE that may not even be exploitable in your environment may cripple a production deployment.

Blindly blocking a container start operation is not the right approach, the policy enforcement mechanism needs to be aware and integrated with the orchestration platform.

The Kubernetes community has accounted for this, and many other, use cases within Kubernetes architecture using admission controllers which allow API requests to the Kubernetes API server to be validated and in some cases modified.

An early implementation of this came in the form of the image policy webhook which allowed Kubernetes to be configured to call a webhook before launching a container. This was available as an alpha feature in Kubernetes 1.4 and higher. Anchore implemented this API natively in the Anchore Engine exposing an HTTPS endpoint against which Kubernetes could issue a webhook and receive an ‘allow’ or ‘deny’ response based on the policy evaluation of the image.
The ImagePolicy webhook remains as an alpha only feature however with Kubernetes 1.8 a new webhook has been added that addresses some of the limitations with the image policy webhook. The new webhook is called ValidatingAdmissionWebhook and was available as an alpha in 1.8 and was promoted to a beta feature in 1.9.

Through a validating admission webhook request, an admission server receives a request that includes the Pod specification before the pod is launched. The admission server can then review the list of images and then call the Anchore API to receive a ‘pass’ or ‘fail’ evaluation of the images to be launched based on the policies configured by the user and stored in the Anchore Engine.

This week Vic Iglesias from Google published a great blog and tutorial that demonstrates how to integrate Anchore Engine with a Kubernetes 1.9 system. What’s great about the blog is that you can follow along and deploy both Anchore Engine and the Admission Controller using Helm and in less than 5 minutes have a Kubernetes Cluster running with image policies secured by Anchore Engine. We look forward to adding the functionality natively into Anchore Engine.

You can read Vic’s blog here and download his code from GitHub, complete with a helm chart.