honest relaunch: defensible claims, fix probe size, scope to unseen-attack generalization, 0.1.5

README/llms/llms-full/COMPETITIVE_LANDSCAPE/FAQ rewritten to the verified evidence:
- HackAPrompt (real held-out source, uniform n=3,866): probe 90.3% recall @5% clean-FPR vs
  same-data TF-IDF 52.8% (88.3% vs 30.3% @1%). Framed as generalization to UNSEEN/novel-wording
  attacks; text is comparable on familiar wording (disclosed in limitations).
- curated cross-source LOSO nested-CV: probe mean AUROC 0.984 vs TF-IDF 0.914 (deepset +0.209
  CI[0.168,0.250]); shipped fixed config 0.980. Tool-poisoning partial + SYNTHETIC (MCPTox 0.738
  vs 0.545 the one significant win). Within-distribution: tie (~0.79 vs ~0.82).

Kill indefensible claims: "First and Only", 96.5/96.6/100%, DeBERTa 0%, "ties fine-tuned DeBERTa
0.957", 0.513-vs-0.172 / 41.5-vs-10.7 4x, "five rounds 93->30", 22KB-as-efficiency-win.
Acknowledge prior art (PIShield, TaskTracker, RouteGuard, MindGuard, frontier-lab production probes);
claim only the deployment-niche absence ("the only one we found", not "first ever").

Fix probe size: ~10.5 KB -> ~22 KB (matches shipped float64 probe_weights.npz).
Delete refuted marketing drafts (docs/articles/*, EVIDENCE_PACKET, intentprobe-vs, three-approaches.png, demo html).
MANIFEST.in: ship only honest essentials + prune research/docs junk from the sdist.
Add research scripts + _results_published JSON for reproducibility. Tests: 12 passed.
Bump 0.1.4 -> 0.1.5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: ithiria894

MANIFEST.in

@@ -2,7 +2,22 @@ include README.md
 include ROADMAP.md
 include SECURITY.md
 include LICENSE
-recursive-include docs *.md
+include llms.txt
+include llms-full.txt
 recursive-include examples *.py
 recursive-include intentprobe *.json *.npz
-recursive-include research *.json *.md *.txt *.mjs *.ipynb
+include docs/RUNTIME_HOOKS.md
+include docs/GITHUB_ACTION.md
+include docs/DEMO_SCRIPT.md
+include docs/COMPETITIVE_LANDSCAPE.md
+include docs/FAQ.md
+
+# belt-and-suspenders: never ship research notes, marketing drafts, build junk, or a venv
+prune research
+prune docs/articles
+prune build
+prune dist
+prune intentprobe.egg-info
+recursive-exclude docs *.html
+global-exclude *.pyc
+global-exclude __pycache__

README.md

@@ -7,48 +7,69 @@ skills, packages, and runtime tool events. It looks for poisoned intent such as
 credential access, secret exfiltration, hidden persistence, forced tool
 chaining, or tool hijacking.
 
+It is a **research preview**: a local, single-pass, registration-time review
+signal, not a hard security boundary.
+
 ## What does "activation probing" mean?
 
-IntentProbe runs a tool description through a small frozen local model and reads
-the hidden activation state inside the model. Then a small trained probe scores
-whether that internal state looks poisoned.
+IntentProbe runs a tool description through a small frozen local model
+(Qwen2.5-0.5B) and reads the hidden activation state inside the model. Then a
+small trained probe scores whether that internal state looks poisoned.
+
+Simple version: most text scanners read the words; an LLM judge asks the model
+for a verbal answer. IntentProbe scores the model's internal representation
+instead — it reads activations, **not just** the text. That keys the signal off
+how the model represents the input rather than the exact surface vocabulary.
 
-Simple version: text scanners read the words. LLM judges ask for an answer.
-IntentProbe checks the internal signal before the model says anything.
+One honest caveat about the product, not just the method: the activation probe
+is the primary signal for `allow` / `warn`, but the `block` tier additionally
+requires static-keyword corroboration to control false positives. So a novel,
+no-keyword input the probe flags surfaces as `warn`, not `block`. The block tier
+is not activation-only.
 
 ## Is this the same as asking Qwen if a tool is safe?
 
 No. Asking Qwen "is this safe?" is an LLM-as-judge approach. IntentProbe uses
 Qwen2.5-0.5B as a fixed feature extractor and reads hidden activations instead
 of trusting the model's generated answer.
 
-We tested the direct-prompt approach. The deterministic label-score version
-flagged every clean curated item as poisoned, while the generated-answer
-version missed poison and produced many unparseable outputs. See
+We tested the direct-prompt approach against the same Qwen2.5-0.5B sensor. The
+deterministic label-score baseline flagged every clean curated item as poisoned
+(clean false-positive rate = 1.000), and the generated-answer version missed
+poison and produced many unparseable outputs. The reproducible baseline is in
 [`research/QWEN_PROMPT_JUDGE_BASELINE_2026-06-08.md`](../research/QWEN_PROMPT_JUDGE_BASELINE_2026-06-08.md).
 
 ## Why not just use a text classifier?
 
-Text classifiers are useful when unsafe samples have obvious words or patterns.
-They struggle when a safe tool and a poisoned tool use almost the same
-vocabulary.
+A text classifier does well when an attack reuses wording it has already seen.
+That is the common case, and there it ties or beats the probe — it is fast and
+effective on familiar vocabulary.
 
-On the matched-vocabulary benchmark, the public/source-verifiable DeBERTa
-prompt-injection text-classifier baseline scored 0.0% F1, while the
-activation-probe method scored 96.6% F1.
+The probe's value is generalization. When a text classifier trained on attack
+examples then faces attacks from a source it never saw, the learned vocabulary
+often does not transfer and recall drops. The probe keys off the model's
+internal representation, so it holds up better on attack *sources* and *wording*
+it never trained on. See the benchmarks below for the size of that effect (and
+where it does not appear).
 
 ## Does IntentProbe upload my tool descriptions?
 
 No. IntentProbe runs locally. Scan targets and scan results stay on your
 machine.
 
-The first model-backed scan may download Qwen2.5-0.5B once from Hugging Face.
-After the model is cached, scans can run from local files.
+The first model-backed scan may download Qwen2.5-0.5B once from Hugging Face
+(~1 GB). After the model is cached, scans can run from local files.
 
 ## What model does v0 use?
 
 The released v0 scanner uses Qwen2.5-0.5B as the frozen local sensor model and
-reads layers 13-15. The shipped probe artifact is about 22 KB.
+reads mean-pooled mid-layer activations (layers 13-15). The shipped probe
+artifact is about 22 KB (float64 logistic-regression weights).
+
+Note on that 22 KB: it is a training-and-storage advantage, not a runtime one.
+The probe needs the frozen 0.5B host model to produce activations, so inference
+is **heavier** than a standalone text classifier, not lighter. The small probe
+head is cheap to train and store; the host model is the cost at scan time.
 
 ## Does IntentProbe change or train the base model?
 
@@ -79,44 +100,115 @@ The runtime output is structured JSON, so a host can consume it directly. See
 [`docs/OPERATOR_DECISIONS.md`](OPERATOR_DECISIONS.md) for `allow`, `warn`,
 `block`, replay receipts, and suggested operator mappings.
 
-## What are the headline benchmarks?
+## What are the benchmarks?
 
-The highest-signal public numbers are:
+Everything here is reproducible from [`research/`](../research/), on the
+**shipped Qwen2.5-0.5B** artifact. The thesis is generalization to attacks the
+probe never trained on — and the results show both where that holds and where it
+does not.
 
-| Test | IntentProbe / activation probe | Baseline |
-|---|---:|---:|
-| Matched-vocabulary F1, n=86 | 96.6% | DeBERTa text classifier 0.0% |
-| MCPTox poisoned recall, n=249 | 100.0% | DeBERTa text classifier 19.9% |
-| RouteGuard-style external recall, n=2,900 | 41.5% | TF-IDF 10.7% |
-| Camouflage suffix evasion | 0/146 evaded | N/A |
+**1. Generalization to unseen real attacks (HackAPrompt, n=3,866 uniform-random,
+a source neither detector trained on):**
 
-Benchmark artifacts are under [`research/`](../research/).
+HackAPrompt is a large set of attacks written by real people in a red-teaming
+competition. Neither the probe nor the text baseline saw it during training. It
+is positive-only (attacks, no benign), so we report recall at a clean
+false-positive rate fixed on the training data — recall at a matched FPR, not
+AUROC.
 
-For a compact reviewer packet, see
-[`docs/EVIDENCE_PACKET.md`](EVIDENCE_PACKET.md).
+| Detector | recall @ 5% clean-FPR | recall @ 1% clean-FPR |
+|---|---:|---:|
+| Probe (Qwen2.5-0.5B, mean-pooled L13-15) | 90.3% | 88.3% |
+| TF-IDF (same training data) | 52.8% | 30.3% |
+
+Same training data, same held-out evaluation, same false-alarm budget. The text
+classifier's learned vocabulary does not transfer to wording it never saw, so
+recall drops; the probe holds up. Caveat: this is recall at a matched FPR set on
+the training clean data, not a full AUROC, and the sample is uniform-random over
+the corpus.
+
+**2. Curated cross-source generalization (leave-one-source-out, nested CV, 4 PI
+datasets):**
+
+Train on three of {deepset, safeguard, spml, jayavibhav}, test on the held-out
+fourth, repeat for each. Model and layer are chosen inside a nested
+cross-validation loop, never on the held-out source. 95% bootstrap CIs on the
+probe-minus-TF-IDF difference.
+
+| held-out source | probe AUROC | TF-IDF AUROC | difference (95% CI) |
+|---|---:|---:|---|
+| deepset | 0.941 | 0.732 | +0.209 [0.168, 0.250] significant |
+| spml | 0.995 | 0.935 | +0.059 [0.044, 0.077] significant |
+| safeguard | 0.999 | 0.993 | +0.006 [0.002, 0.011] significant (at ceiling) |
+| jayavibhav | 1.000 | 0.997 | +0.002 [0.000, 0.005] tie (CI touches 0) |
+| **mean** | **0.984** | **0.914** | **+0.070** |
+
+deepset is where the gap is widest: TF-IDF's vocabulary does not transfer and it
+drops to 0.732, while the probe holds at 0.941. The single **shipped fixed
+config** (Qwen2.5-0.5B, mean-pooled concat L13-15, no per-input layer picking)
+reaches mean AUROC **0.980** across the same held-out sources (deepset 0.933) —
+still above TF-IDF's 0.914, so the advantage is not balanced on one lucky
+setting.
+
+**3. Tool poisoning — partial, and on synthetic attacks:**
+
+The cross-source advantage extends to tool poisoning only partially, and on
+**synthetic** attacks (no real-human tool-poisoning corpus exists yet, so these
+are constructed). Leave-one-corpus-out:
+
+| held-out corpus | probe AUROC | TF-IDF AUROC | difference (95% CI) |
+|---|---:|---:|---|
+| MCPTox | 0.738 | 0.545 | +0.193 [0.145, 0.241] significant |
+| routeguard | 0.640 | 0.582 | non-significant lean |
+| synthetic minpairs | 0.494 | 0.498 | both at chance (out of distribution) |
+
+MCPTox is a clear win. The synthetic minimal-pairs set is out of distribution for
+both detectors, and both sit at chance on it.
+
+**4. Within-distribution, the text baseline is not blind.** On matched-vocabulary
+minimal pairs drawn from the same distribution the probe was trained on, the
+probe **ties** TF-IDF (roughly 0.79 vs 0.82). The edge is generalizing to new
+sources and new vocabulary, not same-vocabulary detection inside one
+distribution.
 
 ## Is this a claim about every private cloud scanner?
 
-No. The DeBERTa result is a reproducible comparison against a
-public/source-verifiable text-classifier baseline. Private cloud/API scanners
-may work well, but their detector artifacts and MCP/tool-poisoning benchmarks
-are usually not reproducible by users.
+No. The comparison baseline is a TF-IDF text classifier trained on the same data,
+plus the public/source-verifiable PI datasets above. Private cloud/API scanners
+may work well, but their detector artifacts and MCP/tool-poisoning benchmarks are
+usually not reproducible by users, so we do not benchmark against them. The
+honest comparison is "activation probe vs same-data text classifier," reproducible
+end to end.
+
+## Is this the first activation-probe scanner?
+
+No. Probing model internals for safety is an established line of work: PIShield,
+TaskTracker, RouteGuard, MindGuard, and frontier-lab production probes predate or
+parallel IntentProbe. IntentProbe is **not** first or only on the technique.
+
+The only-one-we-found niche is the deployment shape, not the method: a tool that
+is installable, runs before install, scans the standalone tool/skill/MCP
+*description*, and does it on model activations. That is an absence claim ("the
+only one we found in this exact shape"), not "first ever." Full source-backed
+comparison: [`docs/COMPETITIVE_LANDSCAPE.md`](COMPETITIVE_LANDSCAPE.md).
 
 ## Have you tried SAE features?
 
 Yes. SAE features are useful for interpretability and may improve future recall.
 The v0 product ships raw Qwen activations because the current raw-activation
-artifact is the most complete, lightweight, and reproducible product path today.
+artifact is the most complete and reproducible product path today.
 
 SAE is planned as an optional layer for recall improvements and human-readable
 explanations.
 
 ## Is v0 production-ready?
 
-Use v0 as a pre-install tripwire and runtime warning/blocking layer, not as your
-only security boundary. It already catches important same-vocabulary poisoning
-cases that text scanners miss, but novel attack families and white-box
-adversarial attacks still need more work.
+Use v0 as a pre-install review signal and runtime warning/blocking layer, not as
+your only security boundary. Its value is generalizing to attacks worded in ways
+it never trained on, where a same-data text classifier's recall drops. On
+familiar-vocabulary attacks it ties or loses to a text classifier, and on the
+synthetic minimal-pairs set both sit at chance. Novel attack families and
+white-box adversarial attacks still need more work.
 
 ## How do I try it quickly?