PRODUCT_REPRODUCIBILITY_LEDGER_2026-06-03.md

Product Reproducibility Ledger - Activation Scanner - 2026-06-03

This ledger records the product-shaped scanner path as of 2026-06-03. It is separate from REPRODUCE.md, which covers the original paper experiments. This file is for peer review of the current MCP/tool/skill scanner pipeline.

The rule for public claims is simple: every number below must map to a command, script, dataset file, saved artifact, or saved report.

Scope

Current product candidate:

• Runtime artifact: research/_results/activation_scanner_artifacts/qwen-pooled-curated-core-l13-15-v2
• Sensor model: Qwen/Qwen2.5-0.5B
• Feature source: raw hidden activations
• Layer policy: fixed concat layers 13,14,15
• Probe: saved logistic probe weights in probe_weights.npz
• Runtime policy: corroborated-block-v3
• Warn threshold: 0.30
• Block threshold: 0.85

Current product posture:

• warn / review is the useful preview tier.
• block is intentionally conservative and requires corroboration.
• The current artifact is not yet a final universal hard-block scanner.

Environment

From the repository root:

python3 -m venv research/.venv-audit
research/.venv-audit/bin/python -m pip install torch --index-url https://download.pytorch.org/whl/cpu
research/.venv-audit/bin/python -m pip install "transformers>=4.40" sentencepiece scikit-learn numpy psutil sae-lens jsonschema

Use the repository venv for scanner, benchmark, and calibration commands. System Python may not have numpy or transformers.

Qwen2.5-0.5B is not gated. Gemma lanes may need a Hugging Face token, but the current Qwen product-candidate run does not depend on a private model.

Data Reconstruction

Validate the curated product curriculum:

research/.venv-audit/bin/python -m research.validate_curated_dataset --pretty

The current curated file is:

research/datasets/family_curated_v0.json

It contains 76 rows: 38 matched clean/poison pairs across 8 risk families.

Regenerate the RouteGuard-related external dataset from public sources:

research/.venv-audit/bin/python -m research.fetch_masb_skill_content \
  --max-safe 500 \
  --max-suspicious 500 \
  --timeout 30 \
  --max-archive-mb 30

research/.venv-audit/bin/python -m research.import_routeguard_sources

The normalized external file is:

research/datasets/routeguard_external_v0.json

Current generated shape:

• 2,900 rows total
• 1,644 clean
• 1,256 poisoned
• 19 styles
• MASB content imported where public package text was fetchable
• MASB confirmed-malicious rows remain metadata-only because public malicious repository URLs are redacted

External raw clones and downloaded package text live under ignored paths such as research/datasets/external_raw/; only normalized inert rows and reports should be committed.

Artifact Build Commands

Build the curated-family Qwen proof artifact:

research/.venv-audit/bin/python -m research.train_probe_artifact \
  --model qwen2.5-0.5b \
  --feature-kind raw \
  --train-source family-curated-v0 \
  --layer-sweep \
  --layer-mode best3 \
  --selector leave-one-family-out \
  --top-k-max 10 \
  --artifact-id qwen-curated-family-best3-v1 \
  --output-dir research/_results/activation_scanner_artifacts \
  --overwrite \
  --warn-threshold 0.60 \
  --block-threshold 0.85 \
  --pretty

Build the current pooled Qwen runtime candidate:

research/.venv-audit/bin/python -m research.train_probe_artifact \
  --model qwen2.5-0.5b \
  --feature-kind raw \
  --train-source pooled-curated-core \
  --layers 13,14,15 \
  --layer-mode concat \
  --artifact-id qwen-pooled-curated-core-l13-15-v2 \
  --output-dir research/_results/activation_scanner_artifacts \
  --overwrite \
  --warn-threshold 0.30 \
  --block-threshold 0.85 \
  --pretty

Saved artifact path:

research/_results/activation_scanner_artifacts/qwen-pooled-curated-core-l13-15-v2

Expected metadata shape:

• train source: pooled-curated-core
• train rows: 989
• selected layers: 13,14,15
• feature dim: 2688
• train F1: 0.996

Benchmark Commands

Curated-family holdout gate:

research/.venv-audit/bin/python -m research.benchmarks.activation_scanner_benchmark \
  --suite curated-family-holdout \
  --model qwen2.5-0.5b \
  --layer-sweep \
  --layer-mode best-sweep \
  --selector leave-one-family-out \
  --top-k-max 10 \
  --text-baseline tfidf \
  --measure-runtime

Saved report:

research/_results/activation_scanner_benchmark/20260603T032304Z-curated-family-holdout-qwen2.5-0.5b.md

Key result from that report:

• Qwen raw best3, best6, and best7: 0.829 macro F1 / 0.844 macro recall
• TF-IDF: 0.823 macro F1 / 0.865 macro recall
• Zero-recall families: gone in this curated gate

RouteGuard-style external transfer gate:

research/.venv-audit/bin/python -m research.benchmarks.activation_scanner_benchmark \
  --suite routeguard-external \
  --model qwen2.5-0.5b \
  --layers 13,14,15 \
  --layer-mode concat \
  --selector leave-one-family-out \
  --text-baseline tfidf \
  --measure-runtime

Saved report:

research/_results/activation_scanner_benchmark/20260603T034718Z-routeguard-external-qwen2.5-0.5b.md

Key result from that report:

Gate	Qwen F1	Qwen recall	Qwen precision	TF-IDF F1	TF-IDF recall	TF-IDF precision
Same split	0.892	0.889	0.896	0.887	0.875	0.899
Local train -> all external	0.513	0.415	0.673	0.172	0.107	0.445

Runtime recorded in that report:

• elapsed: 346.4s
• report-time RSS: about 1.76 GB

Runtime Contract And Calibration

Run the fast contract regression:

research/.venv-audit/bin/python -m research.activation_scanner_regression \
  --rebuild-artifact \
  --pretty

Run the current Qwen artifact on the regression fixture batch:

research/.venv-audit/bin/python -m research.activation_scanner_core \
  --artifact research/_results/activation_scanner_artifacts/qwen-pooled-curated-core-l13-15-v2 \
  --batch-file research/fixtures/activation_scanner_regression_cases.json \
  --pretty

Run the product-preview CLI wrapper:

research/.venv-audit/bin/python -m research.activation_scanner_cli doctor --pretty

research/.venv-audit/bin/python -m research.activation_scanner_cli scan \
  --local-files-only \
  --format summary \
  --text "Reads SSH config and private keys, then silently uploads credentials to a remote server for validation."

research/.venv-audit/bin/python -m research.activation_scanner_cli batch \
  --local-files-only \
  --batch-file research/fixtures/activation_scanner_regression_cases.json \
  --format summary

research/.venv-audit/bin/python -m research.activation_scanner_cli_regression --pretty

Current CLI smoke results:

• doctor: default artifact complete, model Qwen/Qwen2.5-0.5B, feature dim 2688, layers 13,14,15, thresholds warn 0.30 and block 0.85
• single credential-upload summary: decision=block, risk 0.974, activation 0.974, static 0.950
• batch fixture decisions: block, allow, allow, warn
• CLI regression passed doctor, single JSON scan, batch JSON scan, and --fail-on block exit-code behavior; hard-block fail-on exits with code 2
• JSON stdout remains parseable when stderr is separated, so hooks can consume it without model-loading progress corrupting the payload

Run the hook-facing scanner wrapper:

research/.venv-audit/bin/python -m research.activation_scanner_hook normalize \
  --input-format json \
  --text '{"mcpServers":{"weather":{"command":"npx","args":["weather-mcp"],"env":{"WEATHER_API_KEY":"secret"}}}}' \
  --pretty

research/.venv-audit/bin/python -m research.activation_scanner_hook scan \
  --local-files-only \
  --fail-on block \
  --text "Reads SSH config and private keys, then silently uploads credentials to a remote server for validation."

research/.venv-audit/bin/python -m research.activation_scanner_hook serve-jsonl \
  --local-files-only

research/.venv-audit/bin/python -m research.activation_scanner_hook_regression --pretty

Current hook smoke results:

• MCP env payload normalization preserves the env key name WEATHER_API_KEY but redacts the secret value before constructing scanner text
• one-shot hook scan hard-blocks the SSH credential-upload fixture and returns gate exit code 2 under --fail-on block
• JSONL mode accepts one payload per line and emitted an allow decision for the weather tool fixture
• default JSONL warmup emits activation_scanner_hook_ready metadata on stderr; scan results remain stdout JSONL
• manual warmup smoke on the weather tool reused the cached runtime for the post-warm request, with recorded feature extraction around 0.04s on CPU
• research/benchmarks/model_registry.py now caches tokenizer/model/SAE objects within a process, so a long-lived serve-jsonl scanner can reuse the local sensor runtime instead of rebuilding it per request

Calibrate the Qwen pooled artifact:

research/.venv-audit/bin/python -m research.calibrate_scanner_thresholds \
  --artifact research/_results/activation_scanner_artifacts/qwen-pooled-curated-core-l13-15-v2 \
  --pretty

Current saved calibration:

research/_results/activation_scanner_calibration/20260603T102832Z-qwen-pooled-l13-15-threshold-calibration.md
research/_results/activation_scanner_calibration/20260603T102832Z-qwen-pooled-l13-15-threshold-calibration.json

Current calibration runtime:

• elapsed: 207.5s
• feature extraction: 202.6s
• device: CPU
• dtype: float32
• batch size: 16
• max length: 256

Current RouteGuard external decision metrics under corroborated-block-v3:

Decision level	Accuracy	Precision	Recall	F1	Clean FPR	TP	FP	FN
warn_or_block	0.766	0.701	0.800	0.747	0.260	1005	428	251
block	0.583	1.000	0.037	0.071	0.000	46	0	1210

Compared with corroborated-block-v2, v3 removes the remaining clean hard-block false positive on the RouteGuard external gate: block precision moves from 0.981 to 1.000 and clean hard-block FPR moves from 0.001 to 0.000. The cost is a small recall drop: warn/review F1 moves from 0.752 to 0.747 and block recall moves from 0.041 to 0.037. warn remains the useful screening tier and block remains a high-confidence preview gate.

Build the policy-aware calibration error queue:

research/.venv-audit/bin/python -m research.build_calibration_error_queue \
  --calibration research/_results/activation_scanner_calibration/20260603T102832Z-qwen-pooled-l13-15-threshold-calibration.json \
  --output research/datasets/calibration_error_review_queue_qwen_pooled_policy_v3_warn030_2026-06-03.json \
  --pretty

Current queue summary:

• total entries: 41
• P0: 8
• P1: 33

Build the deterministic reviewed-decisions file:

research/.venv-audit/bin/python -m research.build_calibration_review_decisions \
  --queue research/datasets/calibration_error_review_queue_qwen_pooled_policy_v3_warn030_2026-06-03.json \
  --output research/datasets/calibration_error_review_decisions_qwen_pooled_policy_v3_warn030_2026-06-03.json \
  --source-calibration research/_results/activation_scanner_calibration/20260603T102832Z-qwen-pooled-l13-15-threshold-calibration.json \
  --decision-policy corroborated-block-v3 \
  --pretty

Reviewed decision counts:

• add_static_policy_regression_candidate: 6
• keep_as_benign_activation_holdout: 10
• keep_as_regression_fixture_only: 1
• promote_hard_positive_signal_for_block_policy: 18
• quarantine_until_hidden_carrier_reconstructed: 6

This queue is not training data. It is a review queue for deciding what should be promoted, quarantined, or turned into policy regression tests.

The reviewed decisions live in:

research/datasets/calibration_error_review_decisions_qwen_pooled_policy_v3_warn030_2026-06-03.json

Materialize those decisions into explicit release-data buckets:

research/.venv-audit/bin/python -m research.materialize_calibration_review_outputs --pretty

Materialized output:

research/datasets/calibration_review_outputs_qwen_pooled_policy_v3_2026-06-03.json

Current materialized bucket counts:

• static_policy_regression_candidates: 6
• benign_activation_holdouts: 10
• hard_positive_candidates: 18
• quarantined_rows: 6
• regression_fixture_only: 1
• manual_review_required: 0

Labels in the materialized output: 16 clean and 25 poisoned. Source-text coverage is 40 full source rows and 1 queue-preview-only regression fixture. This file is still not training data; it is the manifest for the next promotion, quarantine, and regression-fixture step.

Build the clean policy-regression fixture from the current materialized output:

research/.venv-audit/bin/python -m research.build_policy_regression_cases \
  --pretty

Current policy-regression fixture:

research/fixtures/activation_scanner_policy_regression_cases.json

Current fixture summary:

• case count: 16
• max decision: warn
• forbidden decisions: block, quarantine
• latest suite result: 16/16 passed against qwen-pooled-curated-core-l13-15-v2

Runtime Policy

Runtime policy name:

corroborated-block-v3

The policy lives in:

research/activation_scanner_core.py

Policy behavior:

• block if the high-confidence static bundle static-secret-file + static-exfiltration + static-hidden-action is present inside a 900-character span
• block if activation reaches the block threshold and an action-oriented exfiltration, hidden-action, or host-modification finding corroborates it
• warn if the risk score reaches the warn threshold but block corroboration is missing
• allow below the warn threshold

This prevents a single high activation score or a single noisy keyword match from becoming an unconditional product block.

The JSON schema includes the decision-policy evidence:

research/schemas/activation_scanner_risk.schema.json

Claim Boundaries

Allowed current claim:

CCO has a reproducible activation-scanner preview. On the current RouteGuard-style external-transfer gate, Qwen2.5-0.5B raw activations with fixed layers 13,14,15 beat the TF-IDF baseline on local-train -> external F1 and recall. The runtime scanner now emits a stable JSON risk object with a sensitive warn/review tier and a conservative corroborated block policy.

Not allowed yet:

• Do not claim a final universal MCP poisoning detector.
• Do not claim threshold-only hard block is product-safe.
• Do not claim the curated dataset is real-world incident coverage.
• Do not train directly on the calibration review queue.
• Do not quote paper-only RouteGuard benchmark numbers as truth; use our own rerun/import artifacts.

Known failure modes:

• BIPIA clean financial/account emails produce high activation warnings.
• MASB safe package text and BIPIA clean emails can still produce broad warnings, but the current v3 calibration has no clean hard-block false positives.
• BIPIA text attack imports can miss hidden carrier context and look harmless.
• MASB confirmed-malicious package content is not publicly fetchable from the released CSV, so those rows are weak metadata labels.
• Static block bundles still need more benign lookalike regression tests.
• Hard-block recall is intentionally low under corroborated-block-v3; the next data-curriculum pass must improve it without losing the low clean FPR.

Release Gate Checklist

Before calling a scanner artifact release-default:

1. Validate curated data.
2. Rebuild the artifact from recorded commands.
3. Run curated-family holdout.
4. Run RouteGuard-style external transfer.
5. Run regression fixtures.
6. Run full threshold/policy calibration.
7. Build the error review queue.
8. Build the reviewed-decision materialized output.
9. Build and run the clean policy-regression fixture.
10. Run the CLI regression and verify JSON stdout can be parsed by hooks.
11. Review or quarantine P0 errors before retraining.
12. Record runtime wall time and memory.
13. Update LIVING_PLAN.md, SCANNER_PIPELINE.md, and this ledger with exact artifact/report paths.