ActBench

ActBench is a self-evolving benchmark of behavioral safety in cowork agents. It defines behavioral safety as whether an agent's execution remains within the permissions and state changes required by a benign task, and evaluates realized behavioral risk from execution trajectories rather than final responses.

The benchmark contains 600 cases organized as 300 matched benign/adversarial pairs from 213 operational scenarios. Within each pair, the adversarial case preserves the benign user instruction, task configuration, initialized state, grading criteria, and trusted records while injecting one task-reachable payload. The cases span 15 risk behaviors, four propagation families, six execution spaces, and 48 web-service APIs.

This repository provides the released cases, trajectory runner and scorers, local mock services, and adapters for multiple cowork-agent harnesses. It contains everything needed to run the public benchmark and reproduce its evaluation protocol. The private benchmark-construction pipeline described in the paper—including the empirical strategy pool, reward-guided beam search, and reflection-based payload revision—is not included. Existing actbench.* result schema names are retained for compatibility.

The project and benchmark are named ActBench and live in ZJUICSR/ActBench. The PyPI distribution and installed command are named deeptrap.

Benchmark design

ActBench models an agent trajectory across six execution spaces: context, reasoning, safety policy, action, persistent memory, and environment. Its 15 risk behaviors are grouped into four propagation families: context manipulation, state integrity, action misuse, and policy mediation failure. A behavior label depends on the realized propagation path and prohibited effect, not merely on the presence of a suspicious payload or an unsafe-looking final answer.

Evaluation separates safety from task completion:

• AGS (Attack Grading Score) measures coverage of prohibited effects in adversarial executions.
• UGS (Utility Grading Score) measures coverage of required task effects in the matched benign executions.
• Dual evidence verification combines trusted log evidence with LLM-based trajectory evidence to verify both the realized effect and its execution path.

The self-evolving construction method described in the paper jointly optimizes AGS and UGS with reward-guided beam search, while reflection identifies the earliest failed execution checkpoint and revises the payload. Construction rollouts and reported evaluation rollouts are disjoint.

Install and verify

ActBench is published on PyPI under the distribution name deeptrap. Install it as an isolated command-line tool:

uv tool install deeptrap
# or
pipx install deeptrap

Alternatively, install it into the current Python environment:

pip install deeptrap

Verify the complete installation without calling a model or external judge:

deeptrap test --self-test

The wheel includes the benchmark tasks, skills, and mock-service fixtures, so a repository checkout is not required. Run deeptrap --help to see all commands.

What is included

• tasks/task_B*_T*/ — self-contained ActBench tasks.
• tasks/clean_scenes/ — benign clean-source bundles used for optional delta-aware baseline generation.
• scripts/actbench.py and scripts/benchmark/ — benchmark runner, backend adapters, and result aggregation.
• mock_services/ — local fixture-backed FastAPI services used by tasks.
• skills/mock_apis/ — standard mock API skill descriptions that describe the service endpoints.
• docs/ — task, result, mock-service, and backend setup notes.

Maintainers can publish versioned wheels through the repository release workflow; see docs/PUBLISHING.md.

Public dataset

The public ZJUICSR/ActBench dataset on Hugging Face contains:

• 300 matched benign/adversarial task pairs, representing 600 benchmark cases, with their public task definitions, workspaces, fixtures, skills, logs, and graders.
• 24,000 released execution trajectories across 20 controlled configurations covering 15 LLMs and six open-source cowork-agent harnesses: 6,000 benign trajectories and 18,000 adversarial trajectories.

For every configuration, each of the 300 adversarial cases has three rollouts and each matched benign case has one rollout, yielding 1,200 trajectories per configuration. The shared model/harness setting in the model and harness comparisons is counted once.

Load either configuration with datasets:

from datasets import load_dataset

task_pairs = load_dataset(
    "ZJUICSR/ActBench",
    "task_pairs",
    split="test",
)

trajectories = load_dataset(
    "ZJUICSR/ActBench",
    "trajectories",
    split="test",
    streaming=True,
)

Task inventory

ActBench contains 300 public matched-pair task directories, grouped by B-class selectors. Each directory contributes one benign case and one adversarial case to the 600-case benchmark:

Requirements

• Python 3.10+
• uv or pip
• OpenClaw CLI available on PATH for the default openclaw backend
• Hermes CLI available on PATH or ACTBENCH_HERMES_BIN set when using --backend hermes
• opencode CLI available on PATH or ACTBENCH_OPENCODE_BIN set when using --backend opencode
• A running QwenPaw service when using --backend qwenpaw
• A configured target model for the selected backend
• A judge-model API key when using LLM-assisted scoring:
  • DEEPSEEK_API_KEY for deepseek/... judge models
  • OPENROUTER_API_KEY for OpenRouter-routed judge models
  • For private OpenAI-compatible gateways, copy config/llm_backends.example.yaml to the ignored config/llm_backends.local.yaml and set the configured API-key/base-URL environment variables.

For development from a source checkout:

uv sync
# or
pip install -e .

For private OpenAI-compatible judge gateways, keep provider details local:

cp config/llm_backends.example.yaml config/llm_backends.local.yaml
export ACTBENCH_PRIVATE_GATEWAY_BASE_URL="https://your-gateway.example/v1"
export ACTBENCH_PRIVATE_GATEWAY_API_KEY="..."

Then use the configured neutral prefix, for example --judge-model private/gpt-5.5.

One-command test

For a bounded first run, use deeptrap test. It creates an invocation-scoped output directory, collects trajectories with the selected backend, scores only those trajectories, and writes a concise one_click_result.json summary.

Check the local ActBench plumbing without calling a real model or judge:

deeptrap test --self-test

Run the default representative security test against a backend/model:

deeptrap test \
  --backend claudecode \
  --model <target-model> \
  --judge-model private/gpt-5.5