Using Open Source Tools to Create a Secure Container Based CI/CD Pipeline
Docker gives developers the ability to streamline packaging, storage, and deployment of applications at great scale. With increased use of container technologies across software development teams, securing these images become challenging. Due to the increased flexibility and agility, security checks for these images need to be woven into an automated pipeline and become part of the development lifecycle.
Prior to any implementation, it is important to standardize on a common set of tools that will be critical components for addressing the above requirement. The four tools that will be discussed today are as follows:
Jenkins: Continuous integration tools like Jenkins will be driving the workload for any automated pipeline to run successfully. The three tools below will be used throughout an example development lifecycle. Docker Registry: Docker images are stored and delivered through registries. Typically, only trusted and secure images should be accessible through Docker registries that developers can pull from. Anchore: Anchore will scan images create a list of packages, files, and artifacts. From this, Anchore has the ability to define and enforce custom policies and send the results of these back in the form of a pass or fail. Notary: Notary is Docker’s platform to provide trusted delivery of images. It does this by signing images, distributing them to a registry, and ensuring that only trusted images can be distributed and utilized. Example CI build steps:
- Developer commits code to repository.
- Jenkins job begins to build a new Docker image bring in any code changes just made.
- Once the image completes it is scanned by Anchore and checked against user-defined policies.
- If the Anchore checks do not fail, the image gets signed by Notary and pushed to a Docker registry.
Anchore Policies: As mentioned above, Anchore is the key component to enforcing only secure images progress through next stages in the build pipeline. In greater detail, Anchore will scan images and create a manifest of packages. From this manifest, there is the ability to run checks for image vulnerabilities. Additionally, the ability to periodically check if new vulnerabilities have been published that directly impact a package contained within a relevant image manifest. Anchore has the ability to be integrated with common CI tools (Jenkins), or in an ad hoc manner from a command line. From these integrations, policy checks can be enforced to potentially fail builds. Anchore checks will provide the most value through a proper CI model. Having the ability to split up acceptable base images and application layers is critical for appropriate policy check abstraction. Multiple Anchore gates specific to each of these image layers is fundamental to the overall success of Anchore policies. As an example, prior to trusted base image promotion and push into a registry, it will need to pass Anchore checks for Dockerfile best practices (USER, non ssh open), and operating system package vulnerability checks.
Secondary to the above, once a set of base images have been signed (Notary) and pushed into a trusted registry, it is now a requirement for all ‘application specific’ images to be created. It is the responsibility of whoever is building these images to make sure the appropriate base images are being used. Inheritance of a base layer will apply here, and only signed images from the trusted registry will be able to pass the next set of Anchore policy checks. These checks will not only focus on the signed and approved base layer images, but depending on the application layer dependencies, will check for any NPM or Python packages that contain published vulnerabilities. Policies can be created that enforce Dockerfile and image best practices. As an example, Anchore allows the ability to look for a base image to be in existence via a regex check. These regular expressions can be used to enforce policies specific to image layers, files, etc.
While the above is just an example of how to implement, secure, and enforce images throughout its lifecycle, it is important to understand the differences between tools, and the separate functions each play. Without tools similar to Anchore, it is easy to see how insecure or untrusted images can make their way into registries and production environments. By leveraging gated checks with Anchore, not only do you have control around which images can be used, but teams can begin to adopt core functionality of the other tools outlined above in a more secure fashion.