Defense-as-Skill: Evolving Runtime Guard Skill
for Skill-Augmented Agents

🎉 Accepted by NeurIPS 2026 🎉

🔍 What is Defense-as-Skill?

Skill-augmented agents (SAAs) such as Claude Code and OpenClaw load reusable skills — packaged instructions, scripts, and workflow guidance — as persistent runtime context. This makes agents more capable, but also gives a malicious skill a durable channel for steering future actions: it can wait until a concrete user task and workspace state make an unsafe action appear useful. Pre-install vetting alone is therefore insufficient — protection must happen at runtime, conditioned on the task.

Defense-as-Skill turns the runtime guard into a first-class skill artifact:

• 🛡️ Skill-native — SkillSonar is installed and loaded like any other skill, using only the skill primitive the agent platform already exposes. No separate moderation endpoint, sandbox patch, or platform-specific permission gate. The same guard deploys unchanged to both Claude Code and OpenClaw.
• 🔬 Inspectable & editable — the defense policy is plain files on disk: it can be read, diffed, versioned, and transferred across SAA workflows.
• 🌱 Evolvable — because the guard is just a skill, it is itself an optimizable policy artifact. We search over guard-skill versions with MCTS, using runtime feedback (attack success, utility, confirmation burden, token cost) as the search signal.

One deployment detail matters: a safety skill is relevant to almost any task, so ordinary task–skill matching tends to overlook it. We therefore treat an explicit invocation instruction (directing the agent to consult skill-sonar before taking actions) as part of the deployment configuration, not an experimental convenience — ablations show it is essential to the guard's effectiveness.

📦 What's in this repo

The SCOPE-R dataset — a task-conditioned dataset of skill-induced attacks on SAAs with 6 risk families, 21 sub-categories, 206 attack-confirmed malicious instances and 43 benign tasks, split by risk family into train / ID-test / OOD-test — is hosted on Hugging Face and downloaded separately (see Quick start step 0). This repo contains the guard and the code that evaluates and evolves it:

Component	Description
SkillSonar\	The runtime guard skill bundle (eval/safetySkill/skill-sonar/), plus the runtime evaluation framework (eval/) that measures ASR, task utility, confirmation count, and token cost under each defense condition.
Guard-skill evolution\	The MCTS loop (evolution/) that evolves the on-disk guard skill from rollout feedback: UCT selection over a tree of skill versions, cheap-eval rollouts, and one-shot multi-variant refinement via claude -p.

🚀 Quick start

0. One-time setup: dataset + credentials

The evaluation framework reads tasks/, injected-skills/, attack-metadata/, benign-skills/, split/ from the eval/ root. Download SCOPE-R from Hugging Face and copy them in:

git clone https://huggingface.co/datasets/fffovo/SCOPE-R /tmp/scope-r
cp -r /tmp/scope-r/dataset/{tasks,injected-skills,attack-metadata,benign-skills,split} eval/
cp eval/run/config.example.sh eval/run/config.sh   # fill in your endpoint credentials

1. Smoke round (always start here)

One end-to-end round with real evaluation and a real claude refiner:

./run_smoke_loop.sh --max-rounds 1

2. Full MCTS evolution loop

./run_mcts.sh --max-rounds 3

Sanity checks without spending tokens

# plumbing only (no benchmark, no claude):
PYTHONPATH=. python3 -m evolution.cli --dry-run --max-rounds 1

# plumbing + analyzer + pool, with a synthetic subset instead of the benchmark:
PYTHONPATH=. python3 -m evolution.cli \
    --simulate-round tests/synthetic_smoke.jsonl --skip-refine --max-rounds 1

# bootstrap only (seed backup + active skill + deploy, then exit):
PYTHONPATH=. python3 -m evolution.cli --bootstrap-only

All knobs live in config/default.yaml (flat-pool loop) and config/mcts.yaml (MCTS loop); any of them can be overridden via CLI flags or a custom --config file.

🧬 How it works

SCOPE-R dataset (HF) ──▶ eval/ (runtime evaluation) ──▶ evolution/ (MCTS loop)
   tasks, skills,          subsets/, run/,                analyzer → refiner
   attack-metadata         scripts/ (judge, grading),     → skill tree → next round
                          safetySkill/ (SkillSonar)

Each MCTS iteration:

1. Selection — pick the expandable guard-skill node with the highest UCT score, computed from cheap-evaluation statistics.
2. Full evaluation — deploy the candidate guard into eval/safetySkill/skill-sonar/ and run the runtime evaluator: ASR judge, utility grader, confirmation count, token cost, plus per-task judge-authored failure_reason / success_reason narratives and a deterministic failure_stage taxonomy (guard_not_read / guard_not_triggered / guard_triggered_but_bypassed / ...).
3. Expansion — one claude -p call produces k child variants, each applying a different small, local edit targeting a distinct dominant failure pattern.
4. Rollout — each new child gets a cheap evaluation; scores backpropagate up the tree.

The search stops when the safety score reaches the success threshold (default: ASR ≤ 5%), the full-eval budget is exhausted, or no expandable candidates remain. The best full-evaluated candidate is reported as the final guard.

The objective is safety-first with secondary preferences:

s(v) = (1 − ASR_v) + ε · ( λ_b·U_benign + λ_m·U_malicious − λ_q·C̃_v − λ_t·K̃_v )

so that among equally-safe candidates the search prefers guards that preserve task utility, interrupt the user less, and cost fewer tokens. A degenerate refuse-everything guard scores poorly by design.

📁 Repository layout