Red Hat recently blogged about their progress in adding support for container image signing, a particularly interesting and most welcome aspect of the design is the way that the binary signature file can be decoupled from the registry and distributed separately. The blog makes interesting reading and I’d strongly recommend reading through it, I’m sure you’ll appreciate the design. And of course, code is available online.
Red Hat is, along with the other Linux distributors, well versed in the practice of signing software components to allow end-users to verify that they are running authentic code and is in the process of extending this support to container images. The approach described is different from that taken previously by Docker Inc. however rather than comparing the two approaches I wanted to talk at a high level about the benefits of image signing along with some commentary about trust.
In the physical world, we are all used to using our signature to confirm our identity.
Probably the most common example is when we are signing a paper check or using an electronic signature pad during a sales transaction. How many times have you signed your signature so quickly that you do not even recognize it yourself? How many times in recent memory has a cashier or server compared the signature written with that on the back of your credit card? In my experience that check is likely to happen one out of every ten times and even in those cases that check is little more than a token gesture and the two signatures may not have matched.
That leads me to the first important observation: a signature mechanism is only useful if it is checked. Obviously, when vendors such as Docker Inc, Red Hat, and others implement an image signing and validation system, the enforcement will be built into to all layers, so that in one example a Red Hat delivered image will be validated by a Red Hat provided Docker runtime to ensure it's signed by a valid source.
However it's more likely that the images that you deploy in your enterprise won't just be the images downloaded from a registry, but instead, images built on top of these images or perhaps even built from scratch, so for image signing to provide the required level of security all images created within your enterprise should also be signed and have those signatures validated before the image is deployed. Some early users of image signing that we have talked to have used image signing less as a way of tracking the provenance of images but instead as a method to show that an image has not been modified between leaving the CI/CD pipeline and being deployed on their container host.
Before we dig into the topic of image signing it's worth discussing what a signature actually represents.
The most common example of signatures that we see in our day to day life is in our web browsers where we look for the little green padlock in the address bar that indicates that the connection to the webserver from our browser is encrypted but most importantly it confirms that you are talking to the expected website.
The use of TLS/SSL certificates allows your browser to validate that when you connect to https://www.example.com the content displayed actually came from example.com.
So in this example, the signature was used to confirm the source of the (web) content. Over many years we have been trained NOT to type our credit card details into a site that is NOT delivered through HTTPS.
But that does not mean that you would trust your credit card details to any site that uses HTTPS.
The same principle applies to the use of image signatures. If you download an image signed by Red Hat, Docker Inc, or any other vendor, you can be assured that the image did come from this vendor. The level of confidence you have in the contents of the image is based on the level of trust you already have with the vendor. For example, you are likely not to run an image signed by l33thackerz even though it may include a valid signature.
As enterprises move to a DevOps model with containers we're seeing a new software supply chain, which often begins with a base image pulled from DockerHub or a vendor registry.
This base image may be modified by the operations team to include extra packages or to customize specific configuration files. The resulting image is then published in the local registry to be used by the development team as the base image for their application container. In many organizations, we are starting to see other participants in this supply chain, for example, a middleware team may publish an image containing an application server that is in turn used by an application team.
For the promise of image signing to be fulfilled, at each stage of this supply chain, each team must sign the image to ensure that the 'chain of custody' can be validated throughout the software development lifecycle. As we covered previously those signatures only serve to prove the source of an image, during any point in the supply chain from the original vendor of the base image all the way through the development process the images may be modified. At any step in the supply chain a mistake may be made, an outdated package that contains known bugs of vulnerabilities may be used, an insecure configuration option in an application's configuration file or perhaps secrets such as passwords or API keys may be stored in the image.
Signing an image will not prevent insecure or otherwise non-compliant images from being deployed, however, as part of a post mortem, it will provide a way of tracking down when the vulnerability or bug was introduced.
During each stage of the supply chain, detailed checks should be performed on the image to ensure that the image complies with your site-specific policies.
These policies could cover security, starting with the ubiquitous CVE scan but then going further to analyze the configuration of key security components. For example, you could have the latest version of the apache web server but have configured the wrong set of TLS Ciphers suites leading to insecure communication. In addition to security, your policies could cover application-specific configurations to comply with best practices or to enable consistency and predictability.
Anchore's goal is to provide a toolset that allows developers, operations, and security teams to maintain full visibility of the 'chain of custody' as containers move through the development lifecycle while providing the visibility, predictability, and control needed for production deployment.
With Anchore’s tools, the analysis and policy evaluation could be run during each stage of the supply chain allowing the signatures to attest to both the source of the image and also the compliance of the image’s contents.
In summary, we believe that image signing is an important part of the security and integrity of your software supply chain however signatures alone will not ensure the integrity of your systems.
