Digging into Grafeas
Google recently announced Grafeas, Greek for “scribe”, which is an open source initiative with the goal of standardizing interfaces for auditing and governance, designed for today’s modern software supply chain. I would strongly recommend that you read the blog published by Shopify which covers in detail the use case that Grafeas is designed to address.
There is certainly no shortage of solutions in the market today that address some parts of this problem, whether that is scanning source code for third party code, managing software libraries and their dependencies, license compliance, providing change control and reporting, scanning images for vulnerabilities or secrets, etc.
Here, at Anchore we focus on scanning container images and certifying them using policies defined by our users. There is a lot of data we can extract from the container that we cross reference with external data feeds such as those provided by operating system distributors, package repositories, etc. The data that we collect is exposed through a REST API to allow other systems to integrate with us. However each vendor in this space exposes their own API, with their own data format and standards requiring custom coding and data mapping.
This lack of centralization and standardization makes audit and compliance both costly and complicated. Earlier this year we saw a great example of this problem when the now infamous Apache Struts vulnerability was disclosed. How would an organization know if they were exposed? Was this package present in their Docker images, had those images been run, are they in use now, is this vulnerability present in a virtual machine image deployed on-premises or in the cloud? For many organizations answering this question required querying multiple different platforms using different tools in addition to manual checks and cross-referencing.
Standardizing these APIs and providing a vendor neutral centralized model will facilitate the collaboration and data sharing that’s critical in today’s heterogeneous architecture.
There are two key concepts at the heart of Grafeas’ design: Notes and Occurrences.
A note describes a specific item that can occur in one or more resources while occurrences describe instantiations of a note.
For example we could load notes into Grafeas for all known CVEs. As images are scanned if a CVE is found in an image then an occurrence would be registered showing that note: CVE-2017-5638 was found in image XYZ.
The standard currently describes 6 types of notes, each of which has a strict schema which describes how data should be stored within the note. New note types will be added over time to support a growing number of systems and use cases.
The attestation note is of particular interest. It describes a declaration made by a trusted system known as an Attestation Authority. For example an attestation authority may scan an image for secrets (passwords, keys, etc) and then add an attestation stating that “This image has no stored secrets” or a CI system may add an attestation stating that “This image passed the following tests …”.
Grafeas’ goal is to provide the infrastructure to store and manage metadata about artifacts. As part of the project Google has announced Kritis, Greek for “judge”, which will describe a policy engine that uses the data stored in Grafeas to make decisions on whether a service should be run.
A policy would describe the attestations required for an image to be deployed in a Kubernetes cluster. For example require that the “no secrets”, “tested by CI” and “no critical vulnerabilities present” attestations are present for a given image.
We read the Grafeas announcement with interest, many of the concepts described were very familiar to us based on the work we had done on our own open source analysis and policy engine.
The Anchore Engine is an open source project that provides a centralized service for inspection, analysis and certification of container images. The Anchore Engine is available as a stand-alone open source project that can be deployed on-prem or in the cloud, it also provides the backend analysis and policy engine for Anchore’s Cloud service anchore.io.
The Engine is deployed as a container that stores data in a PostgreSQL database. The engine is built from a number of smaller micro-services that can be deployed in a single container or scaled up separately.
The feed service downloads feed data from Anchore’s cloud which includes CVE data from operating system distributions as well as package data from popular software repositories such as the Node.JS NPM registry and the Ruby Gems registry.
This feed data is stored in the Postgres database and automatically updated at a user defined interval to ensure that the latest vulnerability and package data is used.
The Engine downloads container images from Docker V2 compatible registries and performs a detailed inspection using a set of extensible analysis modules and generates a detailed manifest of the image, a virtual ‘bill of materials’ that includes official operating system packages, unofficial packages, configuration files, language modules and artifacts such as NPM, PiP, GEM, and Java archives. For an example of the level of data collected check out a sample image on Anchore’s Cloud service here: library/centos:latest
In addition to providing information about an image, the Anchore Engine will perform an evaluation of the image based on policies defined by the user. A policy is made up from a set of rules that are used to perform an evaluation a container image. These rules can include checks on security vulnerabilities, package whitelists, blacklists, configuration file contents, presence of credentials in image, manifest changes, exposed ports or other user defined checks. These policies can applied globally or customized for specific images or categories of applications.
A policy can return one of two results:
- Pass: indicating that image complies with your policy
- Fail: indicating that the image is out of compliance with your policy
Users can interact with the Engine using an CLI or a REST API and can be easily integrated into CI/CD pipelines such as Jenkins to ensure that only images pass an organization’s policies are promoted from the CI/CD pipeline.
The Anchore Engine includes a subscription service that can send webhook notifications when imagesare updated, vulnerabilities are detected or an image’s policy status is updated.
The Anchore Engine can be integrated with Kubernetes using the image policy webhook to ensure that only images that meet your policies will be run.
Integrating Anchore Engine with Grafeas
As you can see, there are a number of similarities in the approach taken by Anchore and the Grafeas project, in particular the focus of collecting a broad range of data from an image and then using policies to make pass/fail decisions that can be applied at runtime.
The Anchore database includes the data required by Grafeas, you could almost replace the Postgres Database picture in our architecture diagram with Grafeas.
Our database currently stores:
- Vulnerability Data (CVEs, NVD data, etc)
- Vulnerability to operating system package mapping
- Image metadata
- Package to Image mapping for
- Operating system packages
- NPM packages
- Ruby Gems
- Java Archives
- Python modules
- Other image specific data such as select configuration files and flags for presence of secrets (credentials, certificates, etc)
We created a quick proof-of-concept to see how we can map Anchore’s analysis and policies onto the Grafeas model. As a first step we create a standalone utility that can integrate with an existing Anchore Engine to
- Extract the list of vulnerabilities to import as Notes into Grafeas
- Extract list of packages to import as Notes into Grafeas
- Assemble vulnerability-to-package mappings to import as Occurrences into Grafeas
You can download and run the proof-of-concept code from the following github repo: anchore-grafeas-cli
Adding vulnerabilities to Grafeas shouldn’t be a one time process, CVEs can be added and modified in the database, for example we often see CVEs enter the Debian CVE security database with unknown severity and then be modified to a higher severity as the vulnerability is analyzed and sometimes CVEs are removed based on this analysis. So the vulnerability import will need to be an ongoing process that we run from the engine both to add CVEs and to update existing CVEs.
Using a similar model we can add Notes and occurrences for all the image data that we store, including occurrences of files, Java archives, Ruby Gems, Node.JS NPMs, Python modules and image metadata such as dockerfiles, layers, size, etc.
In addition to simple notes such we are looking at attestations.
These map to the policy checks that we have today in the Anchore Engine.
In the example below, taken from the policy editor in our SaaS service anchore.io, a user is defining checks that they wish to perform against an image including Dockerfile checks and CVE data. Today we create a single policy with multiple checks that is used to evaluate an image.
In the Grafeas model we would likely map single policy checks or groups of policy checks into attestations, for example:
- The image contains no secrets
- The image does not include blacklisted modules
- The image complies with operations team Dockerfile best practices
We still need to spend time digging into Kritis especially to see how attestations are mapped to images. For example allowing an administrator to decide which policies are evaluated based on the image repository and name, or perhaps based on attributes of the image such as the operating system or presence of certain packages allowing distro specific checks, or checks to be performed based on the presence of certain services.
So far we have been talking about adding data into grafeas but having this open API also allows us to read data from Grafeas, for example we could apply a policy check based on data that JFrog Artifactory added to Grafeas to check that artifacts in the image are from an approved source, or we could use the API to query for deployment occurrences to see which images have been deployed and provide a realtime dashboard showing the status of images running on Kubernetes.
We’re excited to engage with the Grafeas community to give feedback on the existing API based on our experience performing image analysis and policy evaluation and also to provide an open source implementation that anyone can use to add compliance and governance to their Kubernetes infrastructure.