When Good Scanners Flag Bad Results
Imagine this: Friday afternoon, your deployment pipeline runs smoothly, tests pass, and you’re ready to push that new release to production. Then suddenly: BEEP BEEP BEEP – your vulnerability scanner lights up like a Christmas tree: “CRITICAL VULNERABILITY DETECTED!”
Your heart sinks. Is it a legitimate security concern requiring immediate action, or just another false positive that will consume your weekend? If you’ve worked in DevSecOps for over five minutes, you know this scenario all too well.
False positives and false negatives are the yin and yang of vulnerability scanning – equally problematic but in opposite ways. False positives cry wolf when there’s no real threat, leading to alert fatigue and wasted resources. False negatives are the silent killers, allowing actual vulnerabilities to slip through undetected. Both undermine confidence in your security tooling.
At Anchore, we’ve been battling these issues alongside our community, and the GitHub issues for our open source scanner, Grype, tell quite a story. In this post, we’ll dissect real-world examples of false results, explain their root causes, and show how vulnerability scanning has evolved to become more accurate over time.
The Curious Case of Cross-Ecosystem Confusion
One of the most common causes of false positives is “cross-ecosystem confusion.” This happens when a vulnerability scanner mistakenly applies a vulnerability from one ecosystem to a different but similarly named package in another ecosystem.
Take the case of Google’s Protobuf libraries. In early 2023, Grype flagged Go applications using google.golang.org/protobuf as vulnerable to CVE-2015-5237 and CVE-2021-22570, both of which affect the C++ version of Protobuf.
As one frustrated user commented in Issue #1179:
“I was just bitten by the CVEs affecting the C++ version of protobuf when I’m using the Go package. Arguably, it shouldn’t even be included on those CVEs in Github because it’s a completely different code base…”
This user wasn’t alone. Looking at the data, we found a whopping 44 instances of these cross-ecosystem false positives across various projects, affecting everything from etcd to Prometheus to kubectl.
The root cause? CPE-based vulnerability matching. The Common Platform Enumeration (CPE) system, while standardized, often lacks the granularity needed to distinguish between different implementations of similarly named software.
When Binary Isn’t So Binary: The System Package Conundrum
Another fascinating case study comes from Issue #2527, where Grype reported CVE-2022-1271 for the gzip utility on Ubuntu 22.04 despite the package being patched.
The problem stemmed from how Linux distributions like Ubuntu handle symbolic links between /bin and /usr/bin. The package manager knew the file was part of the gzip package, but Syft (Grype’s companion tool for generating SBOMs) was identifying the binary separately without connecting it to its parent package.
As Grype contributor Alex Goodman explained during a live stream:
“This issue was related to how Syft handled symlinks, particularly with the ‘user merge’ in some Linux distributions. Syft wasn’t correctly following symlinks in parent directories when associating files with their Debian packages.”
This case is particularly interesting because it highlights the complex relationship between package managers and the actual files on disk. Even when a vulnerability is properly patched in a package, the scanner might still flag the binary if it doesn’t correctly associate it.
The .NET Parent-Child Relationship Drama
.NET developers will appreciate this next one. In Issue #1693, a user reported that Grype wasn’t detecting the GHSA-98g6-xh36-x2p7 vulnerability in System.Data.SqlClient version 4.8.5.
The issue was related to how .NET packages are cataloged. Syft was finding the .NET assemblies and reporting their assembly versions (like 4.700.22.51706), but these don’t align with the NuGet package versions (4.8.5) used in vulnerability databases.
A contributor demonstrated:
$ grype -q dir:.
✔ Vulnerability DB [no update available]
✔ Indexed file system /Users/wagoodman/scratch/grype-1693
✔ Cataloged contents 500f014f33608c18
├── ✔ Packages [1 packages]
└── ✔ Executables [0 executables]
✔ Scanned for vulnerabilities [0 vulnerability matches]
NAME INSTALLED FIXED-IN TYPE VULNERABILITY SEVERITY
System.Data.SqlClient 4.8.5 4.8.6 dotnet GHSA-98g6-xh36-x2p7 HighThis issue highlights the challenges of correctly identifying artifacts across different packaging systems, especially when version information is stored or represented differently.
Goodbye CPE, Hello GHSA: The Evolution of Matching
If there’s a hero in these tales of false results, it’s the shift from CPE-based matching to more ecosystem-aware approaches. In 2023, we published a blog post, “Say Goodbye to False Positives, ” announcing a significant change in Grype’s approach.
As Keith Zantow explained:
“After experimenting with a number of options for improving vulnerability matching, ultimately one of the simplest solutions proved most effective: stop matching with CPEs.”
Instead, Grype primarily relies on the GitHub Advisory Database (GHSA) for vulnerability data. This change led to dramatic improvements:
“In our set of test data, we have been able to reduce false positive matches by 2,000+, while only seeing 11 false negatives.”
That’s a trade-off most security teams would gladly accept! The shift to GHSA-based matching also brought another significant benefit: community involvement in correcting vulnerability data.
Practical Strategies for Managing False Results
Based on our experiences and community feedback, here are some practical strategies for dealing with false results in vulnerability scanning:
- Use a quality gate in your CI/CD pipeline: Similar to Grype’s quality gate, which compares results against manually labeled vulnerabilities, you can create a baseline of known issues to avoid regression.
- Customize matching behavior: Modern vulnerability scanners like Grype allow you to adjust matching behavior through configuration. For instance, you can modify CPE matching for specific ecosystems:
match:
java:
using-cpes: false
python:
using-cpes: true- Create ignore rules for known false positives: When all else fails, explicitly ignore known false positives. Grype supports this through configuration:
ignore:
- vulnerability: CVE-2022-1271
fix-state: unknown
package:
type: binary
version: 18.17.1- Contribute upstream: We believe the best solution is often to fix the data at its source. This is not a consistent practice across the industry. However, as one contributor noted in Issue #773:
“Since we use GHSA now, it’s possible for users to seek to correct the data by raising an issue or PR against https://github.com/github/advisory-database.”
Conclusion: The Never-Ending Quest for Accuracy
The battle against false results in vulnerability scanning is never truly over. Scanners must continuously adapt as software ecosystems evolve and new packaging systems emerge.
The good news is that we’re making substantial progress. By analyzing the closed issues in the Grype repository over the past 12 months, we can see that the community has successfully addressed dozens of false-positive patterns affecting hundreds of real-world applications.
In the immortal words of one relieved user after we fixed a particularly vexing set of false positives: “OMG. This is my favorite GH issue ever now. Great work to the grype team. Holy cow! 🐮 I’m really impressed.”
At Anchore, we remain committed to this quest for accuracy. After all, vulnerability scanning is only helpful if you can trust the results. Whether you’re using our open-source tools like Grype and Syft or Anchore Enterprise, know that each false positive you report helps improve the system for everyone.
So the next time your vulnerability scanner lights up like a Christmas tree on Friday afternoon, remember: you’re not alone in this battle, and the tools are improving daily. And who knows? Maybe it’s a real vulnerability this time, and you’ll be the hero who saved the day!
Are you struggling with false positives or false negatives in your vulnerability scanning? Share your experiences on our Discourse, and report any issues on GitHub. And if you’re looking for a way to manage your SBOMs and vulnerability findings at scale, check out Anchore Enterprise.
