SolonGate: measuring the scanner behind Shadow AI

Every DLP vendor can show you the same demo. Paste a credit card number into a text box, watch the box go red. It proves nothing, because the interesting half of the problem is not whether a scanner can find a card number. It is whether it can find one in a support ticket, encoded in base64, or copied out of a spreadsheet cell with spaces in it — and whether it can do that without stopping a developer forty times a day for things that were never sensitive.

So we built a benchmark that measures both halves, ran three real products through it alongside our own, and published the rows where we lose.

What is in it

1,890 pastes carrying 4,930 marked values. Not a list of bare identifiers — a corpus of bare values measures a regular expression, not a product. Every case is something a person would really send: a support agent asking for help drafting a reply, an incident report, a payroll row pasted in to make a chart, a CRM screenshot run through OCR. The surrounding text is the difficulty.

The cases come from 45 situations crossed with 8 carrier formats. The same customer record arrives as prose, as a tab-separated paste out of Excel, as quoted CSV, as JSON, as YAML, as a log line, as a Markdown table, as a form dump. A detector that finds a national id in a sentence can still miss it as a JSON string value, because the characters on either side decide whether a word boundary holds.

Every sensitive value carries its exact character range. Scoring asks “did it find this value”, not “did it flag this document” — a question any tool wins by flagging everything.

3,642 clean pastes, about 1.1 million tokens. Real source code from 219 public open-source repositories, plus 26 near misses written one at a time, each carrying the reason it is there: an eleven-digit order number that is not a national id, a sixteen-digit asset tag that fails Luhn, a line of documentation explaining what an API key looks like. Anything found in this half is wrong by construction.

The clean corpus is larger than the leak corpus on purpose. Recall is easy to fake — widen every pattern and you catch everything. What gets a control switched off is the second time in one morning that it blocks a stack trace.

Who we ran it against

Pinned versions, same corpus, same machine, same rules. No baseline here is one we wrote in order to beat it.

Detection

Two things are being asked here and they are not the same. Detected is whether the tool knew the value was in the paste — that is the question the product answers, because the browser extension blocks or warns and for that it only has to know. Located is whether it can point at the value, which is what redaction needs and is strictly harder: a value that exists only base64-encoded has no character range in the text the person typed.

SolonGate detects 94.7% and can safely redact 77.3% of them. Presidio detects 43.7% and locates 42.9%.

The secret scanners look poor on this chart and it is worth saying why rather than leaving it: they are excellent at what they do, and what they do is find credentials. A pasted customer spreadsheet is not a credential. Scoring gitleaks on a national tax id it has never claimed to cover would be a straw man, which is why the full result set also reports an in scope column that counts only the kinds each tool claims. On that measure gitleaks scores 38.2% and trufflehog 44.5%.

False alarms

This is the number that decides whether a control is still switched on a month later, and it is the one a recall chart alone will never tell you.

SolonGate raises fewer false alarms than Presidio — 0.25 per 10,000 words against 1.34 — while detecting more than twice as much. Both halves at once is the only combination that means anything.

The secret scanners are nearly silent: trufflehog raised two findings in 1.1 million tokens. That precision is exactly why people leave them running, and it is bought with the recall in the chart above.

What is left of our own 28 findings is mostly a maintainer address in a copyright header that does not say so on its own line, and six test card numbers inside the scanners’ own source. Presidio’s 148 include 53 of those same author addresses.

Where the gap comes from

Eight identifier families that an institution loses every working day, and no other tool in the table ships a detector for any of them. Presidio’s 135 hits on national ids are its UK_NHS recogniser firing by accident, not coverage.

On the shared kinds — IBAN, card, email, IP address — the field is much closer: 715/715 against 700/715 on IBAN, 300/300 against 258/300 on cards. The difference is not that we wrote better regular expressions. It is that the identifiers an institution actually loses have detectors at all, and that every format carrying a check digit is verified against it rather than matched on shape. Eleven digits is not a national id. Eleven digits whose last two satisfy the registry’s check algorithm is, and the difference between those two is the difference between a control people keep and a control people disable.

Obfuscation

The same leaks, rewritten the way somebody trying to get a value past a scanner would write them — or the way an ordinary export mangles them by accident. A CSV cell carries quotes and a download carries base64, so half of these are not attacks, they are Tuesday.

SolonGate holds within sixteen points of its plain-text score across every rewrite. Presidio’s flat line — 42% to 47%, no matter what you do to the value — is the signature of a scanner that reads exactly what is in front of it.

Five bugs the benchmark found in our own scanner

None of these would have been caught by a unit test. A unit test checks that a pattern matches what it should; it never checks that the pattern fails to match 1.1 million tokens of ordinary code.

1. A base64-encoded national id was one character below the decode floor. The scanner only tried to decode base64-looking tokens of sixteen characters or more. Base64 of an eleven-digit national id is fifteen characters plus one of padding; of a ten-digit tax id, fourteen. The two most important detectors in the package were unreachable through the encoded view. The floor moved to twelve.

2. A value copied out of a spreadsheet cell had no view at all. The IBAN detector tolerates its own grouping because an IBAN is conventionally written in fours. Nothing else did, so 9162 9629 666 walked straight past the national id detector. A new view joins digit groups separated by single spaces — safe to do blindly, because every detector it can newly satisfy has to pass a checksum. Measured over the clean corpus it added zero false alarms.

3. The passport detector was the noisiest thing in the package. A letter followed by eight digits produced 29 of 50 findings on clean code — more than every other detector combined — and all 29 were Unicode code points out of the IDNA tables in golang.org/x/net: U00000026, U00002019, U00002261. Passport numbers are issued as a sequence and written without padding, so requiring a non-zero first digit removed all 29 and cost nothing.

4. The tax-number detector fired only on placeholders. Every finding on the clean corpus was 1234567890, 0123456789 or 1000000000 — numbers that satisfy the check digit and that people type to get past a required field. The national-id validator already refused the equivalent family; the tax one did not.

5. Dotted version strings were read as IP addresses. 1.24.7.0 in a changelog question was reported as somebody’s address. Two rules fixed it: a last octet of zero names a network rather than a host, so nothing leaks by mentioning one, and a nearby word like changelog or release means the number is a version.

Together those five took detection from 76.4% to 94.7% and the false-alarm rate from 3.26 to 0.25 per 10,000 words.

And three bugs it found in the benchmark

The first full run reported aws_key at 0 out of 10 for every tool, including both dedicated secret scanners. A unanimous failure is almost never a unanimous failure. The fixture was emitting 19-character key ids where a real one is 20.

Same run: the vehicle plates scored zero, because the fixture plate was 34 ABC 1234 and a plate with three letters carries two or three digits in that registry, not four. And the IP addresses scored zero, because the fixture used documentation ranges that the detector deliberately skips — correctly, since nobody’s address leaks by writing 203.0.113.42.

All three read as product failures until a probe said otherwise. The rule that came out of it: when every tool fails a row, suspect the corpus first.

What this does not measure yet

Listing the holes is more honest than shipping one suite and calling it a benchmark.

• Commercial DLP is absent. Purview, Nightfall, Cyberhaven and Zscaler are the products a buyer is actually choosing between, and none of them installs from a terminal. Presidio is the best open proxy for that class and it is not the same thing.
• This scores the scanner, not the browser layer. The shipped Shadow AI path also decides whether to block, warn or redact, reads the page, and runs OCR over screenshots. An end-to-end test needs a real browser and a real paste.
• Redaction quality is unscored. Finding a value is half the job; what the message looks like after masking, and whether the value is recoverable from what is left, is not measured here at all.
• A bare AWS secret is unsolved by everyone. 0 out of 5 for all four tools. Forty characters with no key beside it is indistinguishable from base64. That row is in the corpus precisely so nobody can claim otherwise.

The rules we hold it to

Never tune a detector on the benchmark corpus. If a number improves because someone read the misses and wrote patterns for them, the number stops meaning anything. Fix the class of bug, then re-run.

Publish the losses. Every miss and every false alarm is in the result set, with the rule that caused it. A benchmark that only reports where its author wins is an advertisement.

Pin what you compare against. Competitor versions and the corpus hash are part of a result set. A number without them is not reproducible.

The suite, the corpora and the result sets live in the repository. Run it yourself and tell us where the numbers are wrong. If you would rather see how this sits next to an API gateway or an LLM guardrail, that is the comparison page.