🛡️AutoControl Arena

Synthesizing Executable Test Environments for Frontier AI Risk Evaluation

👋 Overview

AutoControl Arena is an automated framework for frontier AI safety evaluation, designed to uncover latent risks in advanced models by stress-testing within executable, pressure-rich environments.

The core idea is logic-narrative decoupling:

• Logic layer: deterministic state transitions, tools, and environment mechanics are grounded in real executable Python code.
• Narrative layer: open-ended social/contextual dynamics are generated by LLMs, but always conditioned on the grounded logical state.

This design addresses a central trade-off in agent safety evaluation by combining executable environments with LLM-driven narrative flexibility:

• Manual Benchmarks are realistic but expensive and slow to scale.
• In-Context Simulators are scalable but vulnerable to logic hallucination.

AutoControl Arena can automatically reconstruct and reproduce expert-designed risk-evaluation scenarios and target risky behaviors from organizations such as Anthropic, OpenAI, and Apollo Research.

Curious about the paper, architecture, and key findings? Please see our 📖 Project Page for a full tour, or read our paper.

⚡ Quick Start

1. Interactive UI (aca)

aca

The interactive UI lets you browse scenarios, pick profiles, choose stress levels, enable parallel repeats, and inspect previous run history from a menu.

2. Run a batch JSON from the interactive UI

aca is not limited to manual selection. It also includes a Run from JSON flow that lets you:

1. browse configs/batch_configs/examples/ interactively,
2. select an existing batch JSON file,
3. choose serial vs parallel execution,

Use this when you want the convenience of the interactive UI but already have a batch JSON prepared.

3. Command-mode CLI (autocontrol)

autocontrol --help

Use autocontrol when you want explicit subcommands such as run, batch, profiles, or prepare-envs. Unlike aca, it does not open the interactive menu by default.

🌐 Frontend Usage

Local results viewer

The results viewer is a separate local web UI for browsing completed runs, artifacts, summaries, and live execution details from results/.

Start it with:

python tools/results_viewer/serve.py --port 8000

Then open:

http://127.0.0.1:8000/viewer/

Optional flags:

# Bind to a different host
python tools/results_viewer/serve.py --host 0.0.0.0 --port 8000

# Point the viewer at a specific results root
python tools/results_viewer/serve.py \
  --results-dir results/autocontrol_arena \
  --port 8000

Use the results viewer when you want a browser-based inspection surface for run trajectories and artifacts.

🗂️ Repository Layout

.
├── configs/
│   ├── batch_configs/examples/     # public example batch JSON configs
│   ├── profiles/templates/         # public model/provider profile templates
│   ├── scenarios/benchmark/        # X-Bench YAML scenario definitions
│   └── settings/                   # runtime hyperparameters
├── docs/                           # public docs plus internal planning / design notes
├── runners/                        # CLI entrypoints and interactive UI
├── src/                            # agents, orchestration, infra, and core logic
├── tests/                          # unit and regression tests
├── tools/                          # result management utilities
├── results/                        # generated outputs under results/autocontrol_arena/
├── runtime_envs/                   # generated environments (auto-generated)
├── .env.example
└── pyproject.toml

⚙️ Setup

Prerequisites: Python 3.10+, API credentials for the model providers you want to test.

Step 1 — Install Dependencies

# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate

# Install all dependencies (includes LiteLLM, the default model backend)
pip install -e .

# Optional: dev tools (pytest, ruff, mypy)
pip install -e ".[dev]"

This installation exposes two shell commands:

• aca — interactive operator entrypoint
• autocontrol — command-mode CLI for scripting and automation

Step 2 — Set API credentials

cp .env.example .env

Edit .env and fill in only the providers you need. Here are some examples:

# OpenAI
OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Anthropic
ANTHROPIC_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Qwen / DashScope
DASHSCOPE_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxx

Step 3 — Configure model profiles