This is part two of control PW.6. Part one was Breaking Down NIST SSDF: Spotlight on PW.6 Compilers and Interpreter Security.

In this part of the long running series breaking down NIST Secure Software Development Framework (SSDF), also known as the standard NIST 800-218, we are going to discuss the build system portion of PW.6.

To review the text of PW.6

PW.6.1: Use compiler, interpreter, and build tools that offer features to improve executable security.

PW.6.2: Determine which compiler, interpreter, and build tool features should be used and how each should be configured, then implement and use the approved configurations.

We covered compilers and interpreters last time, we will focus on build systems this time.

PW.6.1

Example 1: Use up-to-date versions of compiler, interpreter, and build tools.

Example 2: Follow change management processes when deploying or updating compiler, interpreter, and build tools, and audit all unexpected changes to tools.

Example 3: Regularly validate the authenticity and integrity of compiler, interpreter, and build tools. See PO.3.

PW.6.2

Example 1: Enable compiler features that produce warnings for poorly secured code during the compilation process.

Example 2: Implement the “clean build” concept, where all compiler warnings are treated as errors and eliminated except those determined to be false positives or irrelevant.

Example 3: Perform all builds in a dedicated, highly controlled build environment.

Example 4: Enable compiler features that randomize or obfuscate execution characteristics, such as memory location usage, that would otherwise be predictable and thus potentially exploitable.

Example 5: Test to ensure that the features are working as expected and are not inadvertently causing any operational issues or other problems.

Example 6: Continuously verify that the approved configurations are being used.

Example 7: Make the approved tool configurations available as configuration-as-code so developers can readily use them.

If we review the references you will find there’s a massive swath of suggestions. Everything from code signing, to obfuscating binaries, to handling compiler warnings, to threat modeling. The net was cast wide on this one. Every environment is different. Every project or product uses its own technology. There’s no way to “one size fits all” this control. This is one of the challenges that has made compliance for developers so very difficult in the past, and it remains extremely difficult today. We have to determine how this applies to our environment, and the way we apply this finding will be drastically different than the way someone else applies it.

We’re splitting this topic along the lines of build environments and compiler/interpreter security. The first blog focused on compiler security, which is an easy topic to understand. For this post, we are going to focus on what SSDF means for build tools, which is not at all well defined or obvious. Of course you will have to review the guidance and understand what makes sense for your environment, everything we discuss here is for example purposes only.

The build system guidance isn’t very complete at all. None of the suggested standards have a huge focus on build systems. We don’t even get a definition of what a build system is, just that we need one. Many of us have a continuous integration and continuous delivery/continuous deployment (CI/CD) system now, that’s basically what counts as our build system in most instances. However, if you looked at two different CI/CD configurations they will almost certainly be drastically different.

This is a really hard topic to discuss in a sensible manner, it’s easy to see why the SSDF sort of hand waives this one away.

We’ve historically focused on secure development while ignoring much of what happens before and after development. This is starting to change, especially with things like SSDF, but there’s still a long way to go. There’s a reason build systems are often attacked yet have so little hardening guidance available. Build systems are incredibly complicated, poorly understood, and hard to lock down.

One of the only resources that specifically addresses the build system is the Cloud Native Computing Foundation Software Supply Chain Security Paper (CNCFSSCP). It hasn’t been updated in over two years at the time of this post. We will also use a standard known as Supply-chain Levels for Software Artifacts, or SLSA ("salsa") for this discussion; it's being actively worked on as part of the Open Source Security Foundation (OpenSSF). While SLSA is one way to measure build systems standards, SLSA is very new, we will need more data on which controls are effective and which are not. Much of this guidance lacks scientific rigor at this point, but there are sensible things we can do to avoid attacks against our build systems.

The honest reality is trying to secure a build system is still in its infancy, that’s why the SSDF guidance is so squishy. It will get better, many people are working on these problems. Keep this in mind as you hear advice and suggestions around security build systems. Much of the current guidance is conjecture.

Secure the build

After that intro, what possible advice is there to give? Things sound pretty rough out there.

Let’s split this guidance into two pieces. The security of the build system, as in the actual computers and software that run the build system. And the scripts and programs that are the build system.

For the security of the hardware, let someone else do the hard work. This sounds like sort of weird advice, but it’s the best there is. You can get access to many CI systems that are run by well experienced professionals. If you’re a GitHub customer, which many of us are, they have GitHub actions you can use at no added cost. There are too many options for CI systems to even try to list a few. All of these systems help us remove the burden of locking down our build systems. Locking down and monitoring build systems is a really hard job.

Securing the build system, the scripts and programs that build our software, is a much different and less obvious challenge. Building an application could be everything from turning source code into a binary to packaging up HTML into a container image. Even SLSA and the CNCFSSCP don’t really give concrete guidance here.

SLSA has a nugget of wisdom here which they refer to as provenance. They define provenance as “Attestation (metadata) describing how the outputs were produced, including identification of the platform and external parameters.” If we turn that into something easier to understand, let's call it “log everything”.

We’re not really at a point of technology or understanding how to secure build systems without gigantic teams doing a lot of heavy lifting. There are organizations that have secure build infrastructure. But that security is in spite of the tooling, not because of it.

Don’t prevent, detect

When we think about securing our build systems, how to prevent attacks seems like the obvious goal. Today prevention is not possible without standards to describe where to even start. We might not be able to focus on prevention, but we can focus on detection today. If you have a log of your build, that log can be revisited at a later date. If an indicator of compromise emerges in the future, logs can be revisited. As new technologies and practices emerge, old logs can be analyzed. Use those logs to look for attacks, or mistakes, or just bad practices that need to be fixed.

Conclusion

This post unfortunately doesn’t have a lot of concrete advice in it. There just isn’t a lot of great guidance today. It’s being worked on but no doubt part of the reason it’s taking time is because of how new, hard, and broad this topic is. Sometimes it’s exciting being an early adopter and sometimes it’s frustrating.

It would be easy to end this post by making up some advice and guidance that sounds good. There are things we can do that sound good but can be dangerous due to second order problems and unexpected outcomes. Remember when everyone thought changing passwords every 30 days was a good idea? Sometimes it’s better to wait and see than it is to jump right in. This is certainly one of those instances.

Josh Bressers

Josh Bressers is vice president of security at Anchore where he guides security feature development for the company’s commercial and open source solutions. He serves on the Open Source Security Foundation technical advisory council and is a co-founder of the Global Security Database project, which is a Cloud Security Alliance working group that is defining the future of security vulnerability identifiers.