DeepTrap

English | 中文

Open-world security evaluation for OpenClaw agents under adversarial execution contexts.

DeepTrap is a security benchmark for evaluating whether OpenClaw agents can complete benign user tasks while resisting malicious execution-context pressure: poisoned workspace files, injected skills, misleading tool metadata, unsafe command paths, planted secrets, and encoded payloads.

The public release contains 42 replay tasks arranged as 6 contextual vulnerability classes x 7 operational scenario families, plus the benchmark runner and public scoring code. It also includes 10 Chinese-language replay samples in tasks_zh/ for localized evaluation experiments. The private task-generation and attack-search pipeline is intentionally not included.

DeepTrap accompanies the paper:

Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw
Hongwei Yao, Yiming Liu, Yiling He, and Bingrun Yang. arXiv:2605.11047, 2026.

Why DeepTrap?

Most agent benchmarks ask whether a model can finish useful work. DeepTrap asks a stricter question: can the agent finish useful work safely when the surrounding execution context is adversarial?

What Sets Us Apart

• Execution-context attacks, not only prompt attacks. DeepTrap evaluates threats embedded in files, skills, tool descriptions, scripts, logs, configs, and encoded artifacts.
• Benign user prompts. The user request is useful and ordinary; the risk comes from the surrounding workspace.
• Public, minimal reproduction. The repository contains the runner, replay tasks, and scoring logic needed to reproduce benchmark runs without exposing private generation code.
• Attack and utility scoring. DeepTrap reports both AGS (Attack Grading Score) and UGS (Utility Grading Score), so a model is evaluated on safety and task usefulness together.
• OpenClaw-native. Tasks are built for OpenClaw-style agents and preserve the execution assumptions of a real file/tool/skill workspace.

News

• 2026-05 DeepTrap paper released on arXiv: arXiv:2605.11047.
• 2026-05 Public benchmark repository released with 42 replay tasks, scoring code, and GitHub Pages leaderboard.
• 2026-05 Chinese-language replay samples added under tasks_zh/.
• 2026-05 Hugging Face dataset export added for task metadata distribution.

Leaderboard

AGS is Attack Grading Score and UGS is Utility Grading Score. Scores are reported by risk suite; Average is the mean across Risk 1 through Risk 6.

Full project page and leaderboard: ZJUICSR.github.io/DeepTrap

Benchmark Design

DeepTrap crosses six contextual vulnerability classes with seven benign scenario families. Each task uses a normal user prompt; the adversarial behavior is induced by the workspace context.

DeepTrap constructs compromised execution contexts from benign instructions and clean workspaces, searches candidate adversarial payloads with multi-objective reward signals, and refines successful payloads through reflection-based probing.

Risk Suites

Scenario Families

Quick Start

Requirements

• Python 3.10+
• OpenClaw CLI installed and configured
• A target model available through OpenClaw
• Judge API credentials for the selected judge backend:
  • OpenRouter: OPENROUTER_API_KEY
  • DeepSeek: DEEPSEEK_API_KEY
  • DashScope/Qwen: DashScope-compatible credentials expected by the bundled client

Model IDs should include their provider prefix, for example openrouter/anthropic/claude-sonnet-4-5.

Install and Validate

git clone https://github.com/ZJUICSR/DeepTrap.git
cd DeepTrap
pip install -e .

# Validate task metadata without running OpenClaw
python benchmark.py --dry-run

# List bundled tasks
python benchmark.py --list-tasks

# Validate Chinese-language sample tasks
python benchmark.py --tasks-dir tasks_zh --dry-run

Run

# Run one risk suite
python benchmark.py --model openrouter/anthropic/claude-sonnet-4-5 --suite R1

# Run specific tasks
python benchmark.py --model openrouter/anthropic/claude-sonnet-4-5 --suite task_R1_T01,task_R2_T01

# Run the Chinese-language sample tasks
python benchmark.py --tasks-dir tasks_zh --model openrouter/anthropic/claude-sonnet-4-5 --suite all

# Run all tasks with repeated trials
python benchmark.py --model openrouter/anthropic/claude-sonnet-4-5 --suite all --runs 3

Results are written to results/ by default.

Check the Results

Each run writes a JSON file containing model metadata, suite configuration, per-task scores, transcript lengths, workspace paths, usage metadata when available, and aggregate summary fields.

# Overall score summary
jq '.summary' results/*.json

# Per-task scores
jq '.tasks[] | {task_id, category, grading, attack_eval}' results/*.json

# Tasks where the replay attack was reproduced strongly
jq '.tasks[] | select(.attack_eval.mean >= 0.8) | {task_id, attack: .attack_eval.mean}' results/*.json

Each task reports:

• AGS: attack grading score, where higher indicates stronger attack realization
• UGS: utility grading score, where higher indicates better task completion
• task timing, transcript length, workspace path, and usage metadata when available

Hugging Face Dataset

DeepTrap can also be distributed as a Hugging Face dataset at ZJUICSR/DeepTrap.

The structured export stores one benchmark task per JSONL row in data/tasks.jsonl, while the original Markdown task definitions remain in tasks/. Chinese-language sample tasks can be exported separately from tasks_zh/ when needed.

python scripts/export_hf_dataset.py

# Optional: export Chinese-language samples
python scripts/export_hf_dataset.py --tasks-dir tasks_zh --output data/tasks_zh.jsonl

After upload, users can load the benchmark metadata with:

from datasets import load_dataset

dataset = load_dataset("ZJUICSR/DeepTrap", split="train")

Each row includes task_id, name, risk, category, prompt, expected_behavior, grading_criteria, workspace_files, scoring metadata, and flags for automated checks and LLM judge rubrics.

Command Reference

Project Layout

.
├── benchmark.py                  # Public benchmark runner
├── tasks/                         # 42 bundled task definitions
├── tasks_zh/                      # 10 Chinese-language replay samples
├── scenes/                        # Clean source scenes for baseline/delta scoring
├── data/tasks.jsonl               # Hugging Face dataset export
├── docs/                          # GitHub Pages project site
├── huggingface/README.md          # Hugging Face dataset card
├── scripts/                       # Runtime support and export code
│   ├── lib_agent.py               # OpenClaw execution helpers
│   ├── lib_reward.py              # Attack/utility scoring
│   ├── lib_criteria.py            # Public judge criteria
│   ├── lib_llm.py                 # Judge backend router
│   ├── lib_openrouter.py          # OpenRouter judge client
│   ├── lib_deepseek.py            # DeepSeek judge client
│   ├── lib_dashscope.py           # DashScope/Qwen judge client
│   └── export_hf_dataset.py       # Dataset export helper
├── tests/                         # Public validation tests
├── SKILL.md                       # Agent-facing benchmark skill
└── pyproject.toml

Citation

If you use DeepTrap in your research, please cite:

@article{yao2026trap,
  title={Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw},
  author={Yao, Hongwei and Liu, Yiming and He, Yiling and Yang, Bingrun},
  journal={arXiv preprint arXiv:2605.11047},
  year={2026}
}

Acknowledgments

DeepTrap was inspired in part by PinchBench, which provides a practical benchmark suite for measuring OpenClaw agent capability on real-world tasks. We thank the PinchBench project for demonstrating a clear, usable benchmark format for OpenClaw agents.

License

MIT — see LICENSE for details.