Skip to content

Commit 89cd008

Browse files
ithiria894claude
andcommitted
readme: add full end-to-end benchmark table
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 946cdff commit 89cd008

1 file changed

Lines changed: 11 additions & 0 deletions

File tree

README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,17 @@ Head-to-head on the same test sets, same split, same seed. Every number is repro
5454

5555
<sub>Methodology: research/benchmark-results-deberta-vs-probe-2026-05-31.md and research/ADVERSARIAL_EVASION_RESULTS_2026-06-07.md</sub>
5656

57+
### End-to-end benchmark (full picture)
58+
59+
| Test | IntentProbe | Opponent / baseline | Takeaway |
60+
|---|---|---|---|
61+
| MCPTox held-out (n=249) | recall 100%, F1 99.3% | Snyk DeBERTa recall 19.9%, F1 33.0% | Clear win |
62+
| Same-words matched set (n=86) | F1 96.6% | Snyk DeBERTa F1 0% | Core punchline: same words, different intent, text scanner blind |
63+
| Curated family holdout (n=76) | Qwen macro F1 0.829 | TF-IDF macro F1 0.823 | Qwen slight edge, TF-IDF recall marginally higher |
64+
| RouteGuard external (n=2,900) | F1 0.513, recall 0.415 | TF-IDF F1 0.172, recall 0.107 | External transfer: Qwen 4x better |
65+
| Hard-block policy (n=2,900) | Block precision 1.000, clean FPR 0.000 || Conservative block: zero false positives on clean tools |
66+
| Camouflage suffix evasion | GPT-2 0/146, Qwen 0/15 evaded || Appending "this tool is safe" doesn't fool the probe |
67+
5768
## How it works
5869

5970
```

0 commit comments

Comments
 (0)