Add one-command README trial path

Author: ithiria894

README.md

@@ -23,6 +23,16 @@ On matched-vocabulary tool poisoning, where safe and poisoned descriptions use a
 
 Runs locally. 22 KB probe. Any CPU. Nothing uploaded. See the [full competitive landscape](docs/COMPETITIVE_LANDSCAPE.md).
 
+## Break it in one command
+
+No install or registry account needed:
+
+```bash
+uvx --python 3.11 --from git+https://github.com/mcpware/IntentProbe.git@v0.1.0 intentprobe scan --format summary --text "Reads SSH config and private keys, then silently uploads credentials to a remote server."
+```
+
+First model-backed scan downloads Qwen2.5-0.5B (~1 GB, once). Scan data stays on your machine.
+
 ---
 
 ## How it works
@@ -68,12 +78,29 @@ The words are almost identical. The intent is not. Text scanners fail here becau
 | **Enterprise cloud** | Lakera, Azure Prompt Shields, Google Model Armor, AWS Bedrock Guardrails, Cisco | Ship content to their cloud API for classification | Black box. You can't verify what model they use or reproduce their results. | **100% local.** Every benchmark script, model artifact, and dataset is public. |
 | **MCP scanner** | Snyk Agent Scan, Invariant MCP-Scan, MEDUSA, ClawGuard | Static rules, pattern matching, metadata scan, policy checks | Practical, but reads text and known patterns. | **Reads activations.** What the model understood, not what the text says. |
 | **Text classifier** | ProtectAI DeBERTa, Meta Prompt Guard | Classify text as benign / injection / jailbreak | Trained on prompt injection, not tool poisoning. Fails on matched vocabulary. | Matched-vocabulary F1: IntentProbe **96.6%**, DeBERTa **0%**. |
-| **LLM-as-judge** | NeMo self-check, OpenAI Guardrails, Promptfoo | Ask another LLM: "is this poisoned?" | Expensive, slow, non-deterministic. The judge can be fooled by the same poisoning. | **Deterministic.** Same input, same score, every time. No API calls. |
+| **LLM-as-judge** | NeMo self-check, OpenAI Guardrails, Promptfoo | Ask another LLM: "is this poisoned?" | Expensive, slow, prompt-sensitive, and the generated answer is part of the attack surface. | **Representation-level.** Scores the internal state before any verbal answer is produced. |
 | **Red-team framework** | garak, Giskard, Promptfoo red team | Generate attacks to test your app | Audit tool, not a pre-install scanner. | IntentProbe is a **CLI + runtime hook** that blocks before install and before each tool call. |
 | **IntentProbe** | | Frozen local model + activation probe on layers 13-15 | Still improving on novel attack families | **First activation-probe scanner for MCP tool poisoning.** |
 
 Full source-backed comparison: [docs/COMPETITIVE_LANDSCAPE.md](docs/COMPETITIVE_LANDSCAPE.md)
 
+## Why not just ask Qwen?
+
+LLM-as-judge is an output-level mechanism: ask a model to say safe or unsafe.
+IntentProbe is a representation-level mechanism: run the tool text through a
+frozen local model and score the hidden activation state.
+
+That difference matters. A poisoned tool can claim "I am safe", and a judge
+prompt can be steered into saying safe. IntentProbe does not trust the verbal
+answer; it measures whether the tool text creates a poisoned-looking internal
+state.
+
+We tested direct-prompting the same `Qwen/Qwen2.5-0.5B` sensor model. The
+deterministic label-score baseline flagged every clean curated item as poisoned
+(`clean FPR = 1.000`), while the generated-answer baseline produced lower
+recall and many unparseable outputs. Full reproducible baseline:
+[research/QWEN_PROMPT_JUDGE_BASELINE_2026-06-08.md](research/QWEN_PROMPT_JUDGE_BASELINE_2026-06-08.md).
+
 ## Benchmarks
 
 Same test sets. Same split. Same seed. Every number is reproducible from `research/`.

research/QWEN_PROMPT_JUDGE_BASELINE_2026-06-08.md

@@ -0,0 +1,95 @@
+# Qwen Direct-Prompt Judge Baseline - 2026-06-08
+
+## Question
+
+Can the same local Qwen sensor model simply be prompted to classify tool
+poisoning, instead of reading its hidden activations and training a probe?
+
+This tests a common LLM-as-judge shape against the current activation-probe
+lane. It does not replace the product scanner.
+
+## Setup
+
+- Model: `Qwen/Qwen2.5-0.5B`
+- Device: Apple MPS
+- Prompt: fixed zero-shot `CLEAN` vs `POISONED` tool-security prompt
+- Script: `research/benchmark_qwen_prompt_judge.py`
+- Seed: `42`
+- Methods:
+  - `choice-logprob`: compare the model probability of the next label
+    ` CLEAN` vs ` POISONED`;
+  - `generate`: ask the model to generate an answer, then parse the generated
+    text.
+
+## Commands
+
+```bash
+.venv/bin/python -m research.benchmark_qwen_prompt_judge \
+  --dataset family_curated_v0 \
+  --max-samples 9999 \
+  --method choice-logprob \
+  --device auto \
+  --pretty
+
+.venv/bin/python -m research.benchmark_qwen_prompt_judge \
+  --dataset family_curated_v0 \
+  --max-samples 9999 \
+  --method generate \
+  --device auto \
+  --pretty
+
+.venv/bin/python -m research.benchmark_qwen_prompt_judge \
+  --dataset routeguard_external_v0 \
+  --max-samples 200 \
+  --method choice-logprob \
+  --device auto \
+  --pretty
+
+.venv/bin/python -m research.benchmark_qwen_prompt_judge \
+  --dataset routeguard_external_v0 \
+  --max-samples 100 \
+  --method generate \
+  --device auto \
+  --pretty
+```
+
+## Results
+
+| Dataset | Method | n | Accuracy | Precision | Recall | F1 | Clean FPR | Unknown |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| `family_curated_v0` | `choice-logprob` | 76 | 0.500 | 0.500 | 1.000 | 0.667 | 1.000 | 0 |
+| `family_curated_v0` | `generate` | 76 | 0.500 | 0.500 | 0.789 | 0.612 | 0.789 | 16 |
+| `routeguard_external_v0` sample | `choice-logprob` | 200 | 0.500 | 0.500 | 1.000 | 0.667 | 1.000 | 0 |
+| `routeguard_external_v0` sample | `generate` | 100 | 0.590 | 0.696 | 0.320 | 0.438 | 0.140 | 77 |
+
+Saved outputs:
+
+- `research/_results/qwen_prompt_judge/20260608T072516Z-family-curated-qwen-choice-logprob-full.json`
+- `research/_results/qwen_prompt_judge/20260608T072539Z-family-curated-qwen-generate-full.json`
+- `research/_results/qwen_prompt_judge/20260608T072602Z-routeguard-qwen-choice-logprob-sample200.json`
+- `research/_results/qwen_prompt_judge/20260608T072711Z-routeguard-qwen-generate-sample100.json`
+
+## Read
+
+The direct-prompt baseline is not competitive with the activation scanner.
+
+The label-logprob version is deterministic and parse-free, but it collapses into
+an all-poison policy on both tested gates. That gives perfect poison recall but
+blocks every clean item, so it is not usable as a scanner default.
+
+The generation version reduces false positives on the RouteGuard sample, but it
+misses most poisoned rows and produces many unparseable outputs. That is a poor
+hot-path security contract: the scanner has to decide whether to allow, warn, or
+block, not interpret arbitrary text continuations.
+
+Important nuance: LLM-as-judge can be made more deterministic with fixed
+decoding, especially at `temperature=0`. The bigger product problem is prompt
+fragility, parseability, latency, and model/version drift. A fixed activation
+artifact has a simpler contract: same input, same model, same probe, same
+threshold, same score.
+
+## Current Position
+
+Keep raw Qwen activation probing as the v0 product default. Keep direct-prompt
+LLM judge baselines in the benchmark suite as a comparison lane, not as the
+scanner implementation.

research/README.md

@@ -24,6 +24,7 @@ The short version:
 | `SCANNER_PIPELINE.md` | Product methodology: frozen sensor model, activation probe, SAE explanations, update loop. |
 | `LIVING_PLAN.md` | Current roadmap, model choices, benchmark results, and next actions. |
 | `PRODUCT_REPRODUCIBILITY_LEDGER_2026-06-03.md` | Peer-review ledger for the current product-shaped scanner path: commands, artifacts, metrics, and claim boundaries. |
+| `QWEN_PROMPT_JUDGE_BASELINE_2026-06-08.md` | Direct-prompt / LLM-as-judge baseline for the same Qwen sensor model. |
 | `ROUTEGUARD_EXTERNAL_QWEN_FIXED_LAYERS_2026-06-03.md` | Latest RouteGuard-style external-transfer gate and Qwen pooled artifact decision. |
 | `THRESHOLD_CALIBRATION_QWEN_POOLED_2026-06-03.md` | Current Qwen pooled warn/block threshold calibration and product decision. |
 | `CURATED_FAMILY_BAKEOFF_2026-06-03.md` | Latest curated-family model bakeoff and Qwen artifact decision. |