When you see a breach in the headlines, it usually reads something like “Known vulnerability exploited to…”Whatever was stolen or broken was compromised because of a bug that had been discovered and fixed by the developers, but not patched in production. Patching is hard.
It’s much harder than most security professionals are willing to admit. It’s not hard because running an upgrade script is hard, patching is hard because without a comprehensive testing suite you never know if an update is going to break your application or systems.
At Tremolo Security, we have already blogged about how we approach patching our dependencies in Unison and OpenUnison. With our release of Orchestra to automate security and compliance in Kubernetes in the past few weeks, we wanted to apply the same approach to our containers. We turned to Anchore’s open source Anchore Engine to scan the containers we publish and make sure they’re kept up to date. In this post we’re going to talk about our use case, why we chose to use Anchore and how we deployed Anchore to scan and update our containers.
Publishing Patched Containers
Our use case for container scanning is a bit different then most. In a typical enterprise you want to have a secure registry with containers that are continuously scanned for known vulnerabilities and compliance with policies. For Tremolo Security, we wanted to make sure that the containers we publish are already patched and kept updated continuously. We work very hard to create an easily patched solution and we want to make sure our customers can feel confident that the containers they obtain from us have been kept up to date.
When we first started publishing containers, we relied on Dockerhub’s automatic builds to publish our containers whenever one of the base images (CentOS or Ubuntu) were updated. This wasn’t good enough for us. The base containers were usually patched once per month, but that was too slow for us. We’d have customers come to us and say “we scanned your containers and there are patches available.” We wanted to make sure that as patches became available they were immediately integrated into our containers.
Why We Chose Anchore
We were first introduced to Anchore a few years ago when they were guests on TWIT.tv’s FLOSS Weekly, a podcast about free and open source software. We had submitted our container for a scan by both Anchore’s service and a well-known provider’s service and received very different results. We tweeted our question to Anchore and they responded with a great blog post explaining how they take into account Red Hat’s updates to CVEs in CentOS for far better and more accurate scan results. That deep level of understanding made it clear to us this was a partner we wanted to work with.
Deploying Anchore
Now for the fun part, we wanted to deploy Anchore’s open source engine on our own infrastructure. We use OKD for various functions at Tremolo Security, including our CI/CD pipeline and publishing. OKD out of the box is far more restrictive than most Kubernetes distributions. It doesn’t allow for privileged containers by default and its use of selinux is very powerful but can be very limiting on containers that are not built to run unrestricted. Finally, OKD doesn’t rely on Helm or Tiller but Anchore does.
Helm and Tiller
I don’t like to deploy anything in my cluster that has cluster-admin access unless I know and am in control of how and when its used. Helm and Tiller have never given me these warm-and-fuzzies and so we don’t use it. That said we needed it for deploying Anchore so we decided to deploy it into Anchore’s project (namespace). When we deployed tiller, we give it a service account that only had administrator access in the Anchore project. As soon as we had Anchore working, we immediately destroyed the Helm and Tiller deployments.
Writing Ephemeral Data
The first issue we ran into was that the containers that makeup Anchore’s engine write ephemeral data into their containers. Most k8s distros will let you do this based on file system permissions, but not OKD. When you need to write ephemeral data you need to update your pods to use empty volumes. We went through each of the deployments in the Helm charts to add them:
- name: service-config-volume mountPath: /anchore_service_config - name: logs mountPath: /var/log/anchore - name: run mountPath: /var/run - name: scratch mountPath: /scratch
and
- name: service-config-volume emptyDir: {} - name: logs emptyDir: {} - name: run emptyDir: {} - name: scratch emptyDir: {}
Unprivileged Containers
Next issue we ran into was that the analyzer needed root access. The other containers thankfully do not. We created a service account and added it to the privileged scc in OKD. SCCs serve the same role in OKD that pod security policies are now serving upstream Kubernetes clusters.
Deleting Helm
Once Anchore was running, did I mention we deleted Helm?
Scanning and Updating Containers
Once Anchore was running, we needed to add our containers and then make sure they were updated appropriately. While the anchore-cli works great for one-off commands, it wasn’t going to scale for us. We publish variants for nearly a dozen containers for Ubuntu, CentOS and RHEL so the cli just wasn’t going to work. The great thing is though that Anchore is cloud-native and the cli just uses an API!
We decided to create a poor man’s operator. An operator is a pattern in the cloud-native world that says “Take all the repetitive stuff admins do and automate it.” For instance, the operators we’re building for OpenUnison and MyVirtualDirectroy will automate certificate management and trusts. Typically operators revolve around a custom resource definition (CRD) and when an instance of a CRD is updated the operator makes sure the environment is brought into line with the configuration of the custom resource. We call this a poor man’s operator because instead of watching a custom resource, we decided to create a CronJob that would run through the containers listed in a CR and if updates are available to call a webhook to being rebuilding.
The great thing about this approach was we could just add new containers to our CR and it would just add them to Anchore’s scans! No fuss, no muss! We’re all open source friends here so we published our code - https://github.com/TremoloSecurity/anchore-os-image-scan.
Closing The Loop
Anchore’s given Tremolo Security a great platform for keeping our containers patched. Knowing when we walk into a customer that they’re scans will give the best possible results is a competitive differentiator for us. We have enjoyed working with Anchore for the last few years and look forward to working with them for many more to come!
