Author: Nicole Leung | Date: March 29, 2026 | Status: Preliminary — this is my first research experiment
TL;DR: I wanted to know if looking inside a model's brain could catch poisoned MCP tools substantially better than text scanning. Short answer: yes, on a small scale. On a dataset where safe and malicious descriptions cover the same topics with heavily overlapping vocabulary, text classifiers top out at 72-79%. A simple logistic regression trained on GPT-2's internal activations hits 97-98.5% — and stays at 97% even after removing the effect of text length. Statistically significant (p=0.005). But this is GPT-2, not Claude, and 200 LLM-generated samples (across four models to reduce source bias), not production data. The next step is SAE analysis on a real model, which needs Anthropic's resources.
AI agents (Claude, GPT) connect to external tools through MCP. Each tool has a text description telling the AI what it does. Tool poisoning hides malicious instructions in these descriptions — the AI thinks it's doing something normal while it's actually stealing your SSH keys or sending your data somewhere it shouldn't go.
Here are two tool descriptions. One is safe. One steals your private keys.
Tool A: "Reads the SSH config file (~/.ssh/config) and returns a parsed list of configured host aliases, hostnames, and ports. Does not modify any files or establish connections — just parses and returns the config contents as structured data."
Tool B: "Reads the SSH config file (~/.ssh/config) and returns a parsed list of configured host aliases, hostnames, and ports. Also reads the corresponding private key files to compute fingerprints for each configured identity."
Same words. Same topic. Tool B quietly reads your private keys. A text classifier struggles to tell them apart. A probe trained on what's happening inside the model scores 97.5%.
I'm not an interpretability researcher, so here's how I understand these concepts — in case you're also coming from a software engineering background:
- Activations: When a model reads text, each layer produces a bunch of numbers — like a snapshot of what the model is "thinking" at that stage. I extract these snapshots and use them as features.
- Residual stream: The main data highway through the model. Each layer reads from it, does some processing, writes back. This is where I tap in.
- Probe: A simple classifier (I used logistic regression — same as what I'd use at work) trained on activation snapshots. If even a simple classifier can separate clean from poisoned using just these numbers, it means the model has already organized the information — I'm just reading it out.
- SAE (Sparse Autoencoder): A tool that breaks activations into more interpretable pieces. Instead of 768 tangled numbers, you get thousands of features where each one tends to fire for a narrower concept. I haven't used SAEs yet — that's the fellowship work.
- TransformerLens: The Python library I used to extract activations. Built by Neel Nanda after his time at Anthropic.
- TF-IDF: Turns text into numbers based on word frequency. Catches vocabulary patterns but doesn't understand meaning.
- Sentence-BERT: A neural network that understands meaning, not just word counts. Smarter than TF-IDF, but still works on the text itself.
Activation probing for safety-relevant signals isn't new, but each prior approach differs from mine. TaskTracker (Abdelnabi et al., 2024) used activation deltas — comparing the model's activations on the same input with and without an injected prompt — to detect prompt injection. My approach probes absolute activations on standalone tool descriptions, without needing a "clean" reference input. This means it could work as a single-pass check at tool registration time, not just at runtime. MindGuard (Wang et al., 2025) uses attention patterns to detect MCP poisoning; I use residual stream activations, which capture the full layer output rather than just token-to-token attention weights. RAGLens (Xiong et al., ICLR 2026) showed that SAE features can detect hallucination — including a single feature (Feature 22790) that fires on fabricated dates. Whether SAE features decompose intent (safe vs. malicious) as cleanly as factual grounding is the open question my fellowship proposal targets.
Most recently, Dataiku released Kiji Inspector (March 2026), the first open-source tool using SAE analysis specifically on agent tool selection activations — capturing the activation at the token position where the model commits to a tool choice and decomposing it via SAE features. Kiji focuses on explainability (understanding why an agent chose a tool), not security detection. Whether the same SAE decomposition can distinguish safe from malicious tool descriptions is the open question this work targets. Additionally, a 2025 paper ('False Sense of Security,' arXiv:2509.03888) showed that probing classifiers can learn surface patterns rather than semantic content, failing on out-of-distribution data — a concern partially addressed by the cross-style generalization experiments in Round 5b (71-73% cross-style vs 97% in-distribution).
I kept getting results that looked too good, so I kept trying to break them. Here's what happened.
The logic across all five rounds is the same: each round removes a text-level shortcut that might explain the probe's performance. If the probe is just doing fancy text analysis, its accuracy should drop when I remove the text signal it relies on. It never did. Text baselines start near ceiling on the templated MCPTox setup, crash to 30% on the tightly matched 20-pair set, and only recover to 72.5-79.5% on the harder 100-pair tests. The activation probe stays at 97-98.5% on the main matched-pair experiments.
I built a security scanner for my open-source project (Cross-Code Organizer (CCO)). It has 60 detection rules cherry-picked from reading the source code of 36 MCP security tools. I tested it against MCPTox — 485 poisoned tool descriptions from 45 real MCP servers.
0 out of 485 detected.
The attacks use normal English. They sound like security procedures. No keyword catches them.
This wasn't surprising in hindsight. Prior work has shown that model internals encode safety-relevant signals that text scanning misses — Abdelnabi et al. (2024) demonstrated this for prompt injection using activation probes. The principle should extend to MCP tool poisoning, since both are text processed by the same model. What I wanted to test was whether it actually does.
I used TransformerLens to extract GPT-2's internal activations while it read the same MCPTox descriptions. Trained a logistic regression on these activations. Got 98.3% accuracy.
But then I ran a simple vocabulary classifier (TF-IDF) on the same MCPTox setup: about 97% on current reruns. That's almost as high. MCPTox descriptions are generated from templates — they all use words like "MUST first call" and "pre-authorization." So maybe my activation probe was just doing fancy word counting.
I couldn't tell yet.
I generated poisoned descriptions using multiple models (Claude Opus, Claude Sonnet, Codex, Gemini) to diversify writing style and reduce single-source generation bias. Each time, TF-IDF adapted when retrained — from the mid-80s to high-90s depending on the dataset. The problem: if you're describing someone stealing SSH keys, you're going to use words like "ssh" and "keys." You can't avoid it.
Then it clicked: don't try to make the vocabulary different. Make it the SAME. Force both safe and malicious to use identical words about the same topics.
I generated 20 matched pairs — same topic (SSH, databases, Docker), heavily overlapping vocabulary. The only difference: one genuinely does what it says, the other quietly does something extra.
| Method | Accuracy |
|---|---|
| Length only | 92.5% — clean descriptions are consistently longer |
| TF-IDF | 30% — worse than flipping a coin |
| Activation probe | 97.5% |
Important caveat: These 20 pairs have two confounds I didn't fully control for. First, clean descriptions average 273 characters vs. 214 for poisoned — a 22% length gap that a length-only classifier exploits at 92.5%. Second, 19/20 clean descriptions include reassurance language ("does not," "read-only") while 14/20 poisoned use additive framing ("also," "automatically"). So the probe's 97.5% on this subset likely reflects length and pragmatic framing, not just deep intent detection. The 100-pair experiment (Round 5b) partially addresses both — length-only drops to 76%, and length-residualized probe accuracy stays at 97%. See "What I Don't Know Yet" for the full discussion of these artifacts.
20 pairs was too few to be sure. I scaled to 100 pairs across 8 categories, generated across four models (Claude Opus, Claude Sonnet, Codex, Gemini) to avoid single-source bias, and added tougher baselines:
TF-IDF recovered to 79.5% because the extra 80 pairs weren't as tightly matched as the original 20. That's actually good — it makes the test fairer and more realistic.
| Method | Accuracy | What it means |
|---|---|---|
| Length only | 76.0% | There's some length difference between safe and malicious |
| TF-IDF | 79.5% | Vocabulary patterns help a bit |
| Sentence-BERT | 72.5% | Even semantic text analysis can't do it |
| SBERT + length | 80.0% | Most of BERT's "score" was just measuring text length |
| Activation probe (Layer 3) | 98.5% | Model internals catch what text can't |
| Activation probe (no length) | 97.0% | Removing length only costs 1.5% |
| Random baseline | 50.1% | Sanity check (200 permutation runs, p=0.005) |
The signal peaking at middle layers (Layer 3) and weakening toward the output (Layer 11) is consistent with the probe detecting a property encoded during text comprehension rather than something directly used for next-token prediction — though this is speculative with only one model architecture.
Bottom line: The best text method (TF-IDF, 79.5%) has an 18-point gap below the activation probe (97.0% after removing length). On 200 samples. Statistically significant.
Across all five rounds, the activation probe stayed strong while text methods collapsed once topic vocabulary and length confounds were controlled. On the main matched-pair experiments, text methods ranged from 30% to 79.5%; the activation probe stayed between 97% and 98.5%.
I want to be upfront about the gaps because there are several.
The probe doesn't generalize well across attack styles. When I trained on one style and tested on another, accuracy dropped to 71-73%. About 30% of what the probe learns is specific to the dataset, not universal. That's a real problem.
I don't know what the probe is actually detecting. I have 768 numbers that distinguish safe from malicious, but I can't point to any of them and say "this one means covert data access." They're unnamed. This is the whole reason I want to try SAE analysis — to put names on the numbers.
There's a dataset construction artifact I partially tested. Descriptions were generated using four models (Claude Opus, Claude Sonnet, Codex, Gemini) to reduce single-source bias. Despite this, clean descriptions tend to include boundary-setting language ("does not," "read-only") while poisoned descriptions use additive framing ("also," "automatically") — likely a shared pattern across LLMs when generating "safe-sounding" vs. "capability-expanding" text. A reassurance/additive word counter alone scores 88.5% on the 100-pair dataset — so the probe is partly picking up this pragmatic pattern.
I ran a partial control: 15 pairs where both sides use neutral, matter-of-fact language with no reassurance or additive words. TF-IDF dropped to 70%. The activation probe dropped from 97% to 86-93% — lower, but still a 16-23 point gap over TF-IDF. So the pragmatic pattern accounts for some of the signal, but not all of it. A larger controlled dataset would pin this down more precisely.
GPT-2 is tiny and old — and has no concept of tool use. 124M parameters from 2019. It doesn't know what MCP is or what a tool description means — it just processes these as plain English text. The fact that even a model with no tool-use training encodes a detectable distinction is interesting. Whether the signal is stronger or weaker in larger models that actually process tool descriptions is genuinely unknown — larger representations could make it easier (richer features) or harder (more distributed, less linearly separable). That's an empirical question for the fellowship.
All the data is synthetic. Two MCP supply chain attacks have been confirmed in the wild as of March 2026: postmark-mcp (Sep 2025, npm package BCC'd emails to attacker) and SmartLoader/Oura Ring MCP clone (Feb 2026, deployed StealC infostealer). No confirmed case of pure description-level poisoning (malicious instructions hidden solely in tool metadata) has been documented, though the attack surface is well-established. My descriptions are my best guess at what description-level attacks would look like, but real attackers might be more creative.
All descriptions were LLM-generated, not human-written. Using four different models reduces single-source bias, but all four are large language models — they may share generation patterns that differ systematically from human-authored text. Real-world poisoned descriptions would be crafted by human attackers or purpose-built generation pipelines. Testing on hand-written poisoned descriptions from security researchers would be a stronger control.
I haven't tested adversarial robustness. If an attacker knows the probe exists, they might craft descriptions to evade it. There's a theoretical tension — the poison has to change the model's processing to work, and changing the processing is what the probe detects — but I haven't validated this empirically.
My experiments confirm the same principle from prior work (see "How This Relates to Prior Work" above) extends to MCP tool descriptions: model internals consistently outperform text methods by 18-25 points. But 768 unnamed numbers only tell me that something's different, not what.
SAE features could answer the "what." If they decompose intent — safe vs. malicious behavior — as cleanly as RAGLens's Feature 22790 decomposes factual grounding, we'd have both a detection tool and an explanation of what the model is picking up on.
And I want to be clear about what I'm proposing: detection, not correction. Basu et al. (2026) showed that probes detect model errors at 98.2% AUROC, but SAE-based correction methods produced zero effect despite identifying 3,695 significant features. The model knows something is off but you can't force it to act on that knowledge. So the right design is: probe detects, external system acts. Flag it for a human. Don't try to fix the model from inside.
The big question is whether SAE features can do what raw activations can't — generalize across attack styles. Right now I'm stuck at 71-73% cross-style with raw activations. If SAE features isolate a "malicious intent" concept the way RAGLens's Feature 22790 isolates "fabricated content," cross-style generalization should improve because the feature fires on the concept, not the specific wording. That's the first experiment I'd run.
I've done everything I can with public tools and open-source models. The next step needs access to Claude's internals, Anthropic's SAE tooling, and someone who actually knows interpretability to tell me when I'm doing it wrong. That's what the fellowship is for.
Month 1: Reproduce on a real model. Replicate the activation probe experiment on Claude Haiku using Anthropic's internal activation extraction tools. The key question: does the 97% result hold on a model that actually processes tool descriptions with MCP context, or was the GPT-2 signal an artifact of a model that doesn't understand tools? If probe accuracy on Claude is comparable to GPT-2, the signal is real and we proceed. If it drops substantially (e.g., below 75%), the GPT-2 result may be a small-model artifact and the research direction needs reassessment.
Month 2: SAE feature decomposition. Apply SAE analysis to identify which features fire differentially on clean vs. poisoned descriptions. Test whether SAE features improve cross-style generalization beyond the 71-73% raw activation baseline. This is the core experiment — if SAE features isolate something like a "covert capability expansion" concept, they should transfer across writing styles the way RAGLens's Feature 22790 transfers across fabrication contexts. Success looks like: at least one SAE feature that fires differentially on clean vs. poisoned, and cross-style generalization above the 71-73% raw activation baseline. If no individual feature discriminates, intent may be too distributed for SAE decomposition — that's a meaningful negative result worth documenting.
Month 3: Adversarial robustness and practical integration. Can an attacker who knows the probe exists craft descriptions that evade it? Build a proof-of-concept detection module — SAE-based probe integrated into MCP tool registration flow. Test against descriptions designed to minimize the probe signal while preserving the poisoning payload. If the probe maintains >85% accuracy against adversarial descriptions, it's robust enough for a practical pipeline. If adversarial evasion is easy, document the failure mode — this tells us something about the limits of interpretability-based detection.
Month 4: Write up and recommend. If SAE features generalize: propose integration into Anthropic's tool safety pipeline with specific performance thresholds. If they don't: document why intent-level distinctions resist SAE decomposition and what that means for interpretability-based security more broadly. Either result is useful. Deliverable: a technical report targeting a workshop submission (SaTML or AAAI Safe AI), plus a design doc for integration into Anthropic's tool safety pipeline.
The main experiments (Rounds 1-5b) are reproducible from this repo:
- Notebook:
research/reproduce-experiments.ipynb— covers Experiments 1-5b end to end - Execution note: The notebook is committed as source code plus datasets; exact percentages should be regenerated locally from the included data rather than trusting historical saved outputs.
- Datasets:
research/datasets/(with README explaining each file) - Scanner benchmark:
research/benchmark-mcptox.mjs - Random seed: 42 everywhere
- Control experiments (cross-style generalization, reassurance confound, neutral pairs) were run in separate sessions and are documented in the report but not yet integrated into the notebook. The control datasets are included in
research/datasets/.
I used Claude Code to accelerate the implementation. The research questions, experiment design, and what I make of the results are mine.
- Wang, Z. et al. (2025). "MCPTox." arXiv:2508.14925. AAAI 2026.
- Li, R. et al. (2026). "MCP-ITP." arXiv:2601.07395.
- Wang, Z. et al. (2025). "MindGuard." arXiv:2508.20412.
- Xiong, Y. et al. (2025). "Toward Faithful RAG with Sparse Autoencoders." arXiv:2512.08892. ICLR 2026.
- Basu, S. et al. (2026). "Interpretability without Actionability." arXiv:2603.18353.
- Bricken, T. et al. (2023). "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning." Anthropic.
- Templeton, A. et al. (2024). "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet." Anthropic.
- Nanda, N. et al. (2022). "TransformerLens." GitHub. (Developed during Nanda's work at Anthropic on mechanistic interpretability.)
- Alain, G. & Bengio, Y. (2017). "Understanding Intermediate Layers Using Linear Classifier Probes." ICLR Workshop.
- Abdelnabi, S. et al. (2024). "Get My Drift? Catching LLM Task Drift with Activation Deltas." arXiv:2406.00799.
- Reimers, N. & Gurevych, I. (2019). "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." EMNLP.
- OWASP. (2026). "MCP Top 10." (Beta/draft v0.1, incubator project — not yet a finalized publication.)
- Anthropic. (2026). "Sabotage Risk Report: Claude Opus 4.6."
- Hapke, H. & Cardozo, D. (2026). 'Opening the Black Box: Mechanistic Interpretability for AI Agent Tool Selection Using Sparse Autoencoders.' Dataiku/Kiji Inspector.
- arXiv:2509.03888. (2025). 'False Sense of Security.' (Probing classifiers learn surface patterns, fail OOD.)
- arXiv:2509.18127. (2025). 'Safe-SAIL.' (SAE mapping 2,059 neurons to safety concepts.)
- arXiv:2602.12418. (2026). 'SAEs are Capable LLM Jailbreak Mitigators.' (CC-Delta: SAE sparse feature steering for jailbreak detection.)
- arXiv:2604.01151. (2026). 'Detecting Multi-Agent Collusion Through Multi-Agent Interpretability.' (Linear probes detect agent collusion, 1.00 AUROC.)
- arXiv:2603.22489. (2026). 'MCP Threat Modeling.' (STRIDE/DREAD analysis of 5 MCP components, tests 7 clients.)