Secret Scanning False Positives: Why They Happen and How to Eliminate Them

Introduction

Secret scanning has an odd problem. The tooling is good. Bots scan public GitHub commits within seconds of a push, and AWS has reported that exposed access keys often get exploited within minutes. Detection is fast, cheap, and everywhere. Yet organizations still leak credentials that stay live for months.

The scanner usually isn't the thing that failed. It found the secret, raised an alert, and that alert got lost among thousands of others that turned out to be nothing. Surveys of security operations teams keep finding the same thing: most alerts are never investigated. When a tool raises the alarm often enough for nothing, people stop reacting, and the one alert that actually mattered slips past with the rest.

For secret scanning, that noise isn't a minor annoyance. It's how real leaks stay hidden. GitGuardian detected over 12 million new exposed secrets on public GitHub in a single year, and no team can work through that volume by hand when much of it is junk. So reducing false positives isn't about a tidy dashboard. It's about making sure the findings that matter actually get looked at.

This guide walks through what a false positive really is, the two root causes behind most of them, an over-correction that quietly makes things worse, and a layered approach that turns a noisy scanner into one your team will trust.

Two Kinds of False Positive

To cut false positives you first have to be clear about what they are, because the term covers two very different failures with two very different fixes.

The first kind isn't a secret in the first place. The scanner flagged a string that looks like a credential but never was one: a UUID that resembles an API token, a git commit SHA, a base64-encoded image fragment, or a placeholder like AKIAIOSFODNN7EXAMPLE copied straight out of the AWS docs. The finding is wrong about what the string even is.

The second kind is trickier, and more dangerous to get wrong. The string really is a credential, but it can't hurt you: a revoked key, an expired token, a throwaway credential for a local test fixture, a demo key scoped to a sandbox with no real access. The match is right. The risk call is wrong.

These two belong to different parts of the pipeline. The first is a detection problem, meaning your matcher is too loose. The second is a validation problem, meaning you found something real but never checked whether it still works. Most false-positive work is really these two jobs under one name, and teams that treat them as a single sensitivity dial to turn down tend to bury real secrets along with the noise.

Root Cause #1: Pattern Matching Without Boundaries

Most secret detection begins with regular expressions. A Stripe live key is sk_live_ followed by a fixed run of alphanumeric characters, and an AWS access key ID starts with AKIA and 16 more. Patterns like these are tight and well-anchored, so they rarely misfire.

The trouble is the long tail of credentials with no distinctive prefix. To catch those, rules fall back on keyword matching: look for strings near words like token, secret, apikey, or a vendor name. Keyword rules are where precision tends to break down.

Take a rule meant to catch a service whose keyword is, say, audd. Written lazily as a case-insensitive substring, it also matches inside fraudDetection, audit_log, and inaudible. Each of those is a false positive, and a large codebase holds thousands of them. The fix is tiny, just a word boundary in the expression, but the rule never had one. Do that across a few hundred keyword rules and the scanner starts flagging a real chunk of ordinary source code.

It gets worse the moment you import rulesets from open-source scanners. TruffleHog, Kingfisher, and similar projects ship huge rule collections built by many contributors over many years. They are an excellent starting point, but they were written to different standards, and plenty of their keyword patterns have no boundaries or context constraints. Import a thousand community rules as-is and you inherit a thousand authors' ideas of what "close enough" means. The engine isn't broken. The rules were just never checked for the false positives they produce at scale.

Entropy analysis, the other half of pattern-based detection, has its own failure mode. High-randomness strings are a reasonable signal for secrets, but they also describe UUIDs, content hashes, minified JavaScript, and base64 blobs. A raw entropy threshold can't separate a leaked private key from a compiled asset: too strict and it misses real secrets, too loose and it flags every hash in your lockfiles.

The takeaway from the first root cause is that pattern quality isn't something you get for free. Boundary-aware expressions, context requirements, and actually checking what each pattern matches in real repositories are what separate a usable signal from a flood.

Root Cause #2: Detection Is Not Validation

Now the more important half, and the part most tools get wrong.

Say your pattern matching is perfect. You find a string that is unmistakably shaped like a Stripe secret key. What do you actually know? Only that it looks like one. You don't yet know if it's live, revoked, expired, a test key, or a fragment someone pasted into a comment as an example. A match tells you a string resembles a credential. It tells you nothing about whether that credential is dangerous right now.

The only way to close that gap is active validation: take the candidate and ask the issuing service whether it works. This is also where the second, quieter kind of false positive comes from, because validation done badly is barely better than none.

The usual mistake is to trust the HTTP status code. You send the key to the vendor's API, you get a 200 OK, and you mark it valid. But plenty of APIs return 200 for requests carrying an invalid or unrecognized credential. Some return 200 with an error object in the body. Some have public endpoints that answer 200 no matter what you send. A verifier that reads only the status line will confirm dead keys as live, and get it wrong in the other direction just as often. Real validation has to read the response body and check for the shape a genuine authenticated response takes for that specific service, not just the status code the transport returned.

This is the biggest single lever on how much you can trust a scanner, and it's the one most teams never pull, because building and maintaining response-body validators for hundreds of services is real, continuous work. The payoff is worth it: it clears out the entire "real string, but dead" category. A finding validated against the live service with a body-level check isn't a guess that something looks risky anymore. It's a confirmed, working credential, and that difference is what everything else rests on.

The Over-Correction Trap

This is where well-meaning teams make it worse.

Facing a noisy scanner, the reflex is to turn sensitivity down: raise entropy thresholds, disable the chatty rules, add broad allowlists. Push that far enough and you've swapped false positives for false negatives, which is a worse deal. A false positive costs a few minutes of triage. A false negative costs you a breach. A quiet dashboard you got by suppression isn't security. It's the same exposure with the warning lights switched off.

The second over-correction is more dangerous, and it shows up in automated remediation. Once a team trusts its pipeline enough to act on its own, revoking a flagged credential, deleting it, rotating it, the cost of a wrong call goes way up. And a lot of automation quietly treats "I couldn't verify this" as "this is invalid." A verification rule gets renamed or deleted, or a service has no validator at all, so the check returns a bare failure, and the pipeline reads that as "dead credential" and revokes it. Except the credential was never dead. It was unverifiable, and the automation just killed a live key in production.

The fix is a rule that sounds obvious and gets broken constantly: a verification has three outcomes, not two. Valid, invalid, and indeterminate. When the service confirms a credential is bad, it's invalid and you can act. When you simply couldn't check it, because the validator is missing, the network dropped, or the rule changed, it's indeterminate, and indeterminate must never be treated as invalid. Don't revoke, delete, or disable on an inconclusive result. Surface it to a human and retry it, but don't let an operational gap pose as a security verdict. Treating "we don't know" as "it's dead" is how a false-positive cleanup ends up causing its own outage.

A Layered Approach to Cutting False Positives

No single setting fixes this. Detection you can trust comes from stacking filters, each aimed at a specific kind of false positive, so a finding has to clear every layer before it reaches a person.

Boundary-aware, context-aware patterns. Keyword rules should require word boundaries and, where possible, some structural context, an assignment or a known key format nearby, rather than a bare substring. That kills the fraudDetection-matches-audd class at the source.

Calibrated, type-aware entropy. Treat entropy as one signal among several, and exclude the strings that are high-entropy by nature: UUIDs, commit hashes, known asset formats, dependency lockfiles.

Path and file context. A key in test/fixtures, examples/, a vendored dependency, or a doc snippet warrants different handling than one in production config. Context isn't a reason to ignore a finding, but it's useful signal.

Curated allowlists for known-benign values. Documentation example keys and shared sandbox tokens belong in an explicit, auditable allowlist, not a rule you quietly loosened.

Active, body-level validation. The decisive layer. Every candidate that clears pattern filtering gets checked against the issuing service with a response-body validator, which separates "looks like a credential" from "is a working credential." It removes the real-but-dead class that pattern filtering can't touch at all.

Confidence scoring instead of a binary flag. A well-formed, high-entropy finding in a production path that validated as live shouldn't carry the same weight as a low-entropy string in a test file that failed validation. Ranking by confidence lets a team spend attention where it matters and keeps indeterminate results in view without burying the true positives.

The layers are complementary, not redundant. Boundaries handle the "not a secret" class, validation handles the "dead secret" class, and the three-state rule keeps remediation from causing new incidents. Drop any one of them and its particular kind of false positive comes back.

How Cremit Approaches It

Cremit's detection is validation-first for exactly this reason. Pattern matching decides what's worth checking, but it never decides what gets reported. Every candidate credential is validated against the live service with response-body checks rather than status codes, so what reaches your team isn't a string that resembles a secret but a credential confirmed to work. Anything that can't be conclusively verified is surfaced as indeterminate and never actioned silently, which stops automated response from revoking a key it never confirmed was dead. What you get is a queue you can act on, where an alert means a real, working, exposed credential, and the findings that matter aren't buried under everything that only looked alarming.

Conclusion

False-positive reduction comes down to trust. A scanner that's right often enough gets acted on. One that's wrong often enough gets ignored, and an ignored scanner protects nothing, however good its detection engine is. Getting from noise to signal takes better pattern hygiene, real validation against the live service, and the discipline never to read "we couldn't verify this" as "this is safe to kill." Do that, and the alert queue stops being a place where real breaches hide and becomes the place where they get caught.