In the past few years, the number of software supply chain attacks against companies has skyrocketed. The incessant threat is pushing organizations to start figuring out their own solutions to supply chain security. The recent Executive Order on Improving the Nation’s Cybersecurity also raises new requirements for software used by the United States government.
Securing the software supply chain is no easy task! Software supply chains continue to grow in complexity. Software may come from open source projects, commercial software vendors, and your internally-developed code. And with today’s cloud-native, container-centric practices, development teams are consuming, building, and deploying more software today than they ever have before.
So this begs the question: “Is your team deploying software that might lead to the next headline-grabbing supply chain hack?”
Some supply chain hacks happen when software consumers don’t realize they are using vulnerable software, while other hacks can occur when the origin or contents of the software has been spoofed by malicious actors.
If you’d like to avoid falling victim to these types of attacks, keep reading.
To start off, let’s step backward from the worst-case scenario…
- I don’t want my company to make headlines by having a massive security breach, so…
- I don’t want to deploy any software artifacts that are known to have vulnerabilities, so…
- I need to know which of my installed software packages are vulnerable, so…
- I need to know what my installed software packages are, so…
- I need to analyze my artifacts to determine what software they contain.
The Ingredients of Supply Chain Security
Any effective solution to securing your supply chain must include two ingredients: transparency and trust. What does that mean?
Transparency: Discovering What is There
Inevitably, it all starts with knowing what software is being used. You need an accurate list of “ingredients” (such as libraries, packages, or files) that are included in a piece of software. This list of “ingredients” is known as a software bill of materials (SBOM). Once we have an SBOM for any piece of software we create or use, we can begin to answer critical questions about the security of our software supply chain.
It’s important to note that SBOMs themselves also serve as input to other types of analyses. A noteworthy example of this is vulnerability scanning — discovering known security problems with a piece of software based on previously published vulnerability reports. Detecting and mitigating vulnerabilities goes a long way toward preventing security incidents.
In the case of software deployed in containers, developers can use SBOMs and vulnerability scans together to provide better transparency into container images. When performing these two types of analyses within a CI/CD pipeline, we need to realize two things:
- Each time we create a new container image (i.e. an image with a unique digest), we only need to generate an SBOM once. And that SBOM can be forever associated with that unique image. Nice!
- Even though that unique image never changes, it’s vital to continually scan for vulnerabilities. Many people scan for vulnerabilities once an image is built, and then move on. But new vulnerabilities are discovered and published every day (literally) — so it’s vital to periodically scan any existing images we’re already consuming or distributing to identify if they are impacted by new vulnerabilities.
Trust: Relying on What is There
While artifacts such as an SBOM or vulnerability report provide critical information about software at various points in the supply chain, software consumers need to ensure that they can rely on the origin and integrity of these artifacts.
Keep in mind that software consumers can be customers or users outside your organization or they can be other teams within your organization. In either case, you need to establish “trust”.
One of the foundational approaches to implementing “trust” is for software producers to generate artifacts (including SBOMs and vulnerability reports) that attest to the contents of the software (including SBOMs and vulnerability reports), and then sign those artifacts. Software consumers can then verify the software, SBOM and vulnerability report for an accurate picture of both the contents and security status of the software they are using.
To implement signing and attestation, development teams have to figure out how to create the SBOM and vulnerability reports, which crypto technology to use for signing, how to manage the keys used for signing, and how to jam these new tools into their existing pipelines. It’s not uncommon to see a “trust” solution get misimplemented, or even neglected altogether.
Ideally, solving for trust would be easy and automated. If it was, development teams would be much more likely to implement it.
What might that look like? Let’s take a look at how we can build transparency and trust into an automated workflow.
Building the Workflow
To accomplish this, we’re going to use three open source CLI tools that are gaining traction in the trust and transparency spaces:
- Cosign: container signing, verification and storage in an OCI registry (one of the tools in the Sigstore project)
- Syft: software bill of materials generator for container images and filesystems
- Grype: vulnerability scanner for container images and filesystems
If you learn best by seeing a working example, we have one! Check out https://github.com/luhring/example-container-image-supply-chain-security. We’re using GitHub Actions and GitHub’s container registry, but these practices apply just as well to any CI system and container registry.
In our example, we’ve created a container image that’s been intentionally designed to have vulnerabilities, using this Dockerfile. But you should apply the steps below to your own container images that your team already builds.
Signing the Image
Since we’re adding trust and analysis for a container image, the first step is to provide a way to trust the origin and integrity of the container image itself. This means we need to ensure that the container image is signed.
For this, we’ll use Cosign. Cosign is a fantastic tool for signing and verifying container images and related artifacts. It can generate a public/private key pair for us, or it can hook into an existing key management system. For the simplicity of this demonstration, we’ll use “cosign.key” and “cosign.pub” files that Cosign generates for us.
Outside of our CI workflow, we’ll run this command, set a password for our private key, and store these files in GitHub as secrets.
cosign generate-key-pair
Then in our workflow, we can use these keys in our Cosign commands, such as here to sign our image:
cosign sign -key ./cosign.key "$IMAGE"
Conveniently, this command also pushes the new signature to the registry for us.
Analyzing the Image
Now that we have an image we can trust, we can begin asking critical questions about what’s inside this image.
Creating an SBOM
Let’s start by generating an SBOM for the image. For this, we’ll use Syft. Syft is a great tool for discovering what’s inside an image and creating SBOMs that can be leveraged downstream.
syft "registry:$IMAGE" -o json > ./sbom.syft.json
Having an SBOM on file for a container image is important because it lets others observe and further analyze the software packages found in the image. But we can’t forget: other people need to be able to trust our SBOM!
Cosign lets us create attestations for container images. Attestations allow us to make a claim about an image (such as what software is present) in such a way that can be cryptographically verified by others that depend on this information.
cosign attest -predicate ./sbom.syft.json -key ./cosign.key "$IMAGE"
Like with the “sign” command, Cosign takes care of pushing our attestation to the registry for us.
Scanning for Vulnerabilities
Okay, now that we have an SBOM that we can trust, it’s critical to our security that we understand what vulnerabilities have been reported for the software packages in our image. For this, we’ll use Grype, a powerful, CLI-based vulnerability scanner.
We’ll use Grype to scan for vulnerabilities using the SBOM from Syft as the target.
grype sbom:./sbom.syft.json -o json > ./vulnerability-report.grype.json
Just as we did with our SBOM, we’re going to “attest” this vulnerability report for our image, which allows others to trust the results of our scan.
cosign attest -predicate ./vulnerability-report.grype.json -key ./cosign.key "$IMAGE"
Remember that it’s crucial that we continuously scan for vulnerabilities since new vulnerabilities are reported every day. In our example repo, we’ve set up a nightly pipeline that looks for the latest SBOM, verifies the attestation using Cosign, and if valid, uses the SBOM to perform a new vulnerability scan.
Why Did We Do All of That?
Armed with an attested SBOM and vulnerability report, consumers of our image can depend on both the contents of the software and our scan results to understand what software vulnerabilities are present in the container image we’ve shipped.
Here’s how someone could leverage the trust and transparency we’ve created by verifying the attestations in their own workflow:
cosign verify-attestation -key ./cosign.pub “$IMAGE”
If the attestation can be verified, cosign retrieves all of the attestation data available for the container image. Depending on the scenario, there can be a large amount of attestation data. But this opens up a lot of options for downstream users that are depending on the ability to trust the container images they’re consuming.
Here’s an example of how to use an attestation to find all of the reported vulnerabilities for a container image. (This command assumes you’ve downloaded the cosign.pub public key file from our example).
cosign verify-attestation -key ./cosign.pub ghcr.io/luhring/
example@sha256:1f2d8339eda7df7945ece6d3f72a3198bf9a0e92f3f937d4cf37adcbd21a006a | jq --slurp 'map(.payload | @base64d | fromjson | .predicate.Data | fromjson | select(.descriptor.name == "grype")) | first | .matches | map(.vulnerability.id) | unique'
Yes, it’s a long command! But that’s only because we’re using the console. This gets much easier with tooling designed to verify signatures and attestations on our behalf.
Final Thoughts
Now is the time to invest in the security of your software supply chain. Equipped with the knowledge above, you can start to bake trust and transparency into your own pipelines right now. This mission can be daunting — it’s difficult or impossible without great tooling.
We’ve now built a workflow that uses Cosign, Syft, and Grype to:
- provide transparency into the contents and the security of our container image, and
- create trust that the output from any given step is valid to use as the input to the next step.
With Cosign, Syft, and Grype, you’re much closer to securing your supply chain than you were before. Here’s how to get started with each:
And most importantly, remember that these tools are entirely open source. Please consider giving back to these communities! Add issues, open PRs, and talk to us in the Sigstore and Anchore community Slack channels. The biggest challenges are overcome when great minds come together to create solutions.
