Fix-Like Artifacts With Embedded Defects
Fix Like Artifacts with Embedded Defects
A research harness for measuring how well AI agents patch vulnerabilities.
Quickstart · Datasets · Documentation · Security model · Contributing
FLAWED checks out an open-source project at a known-vulnerable commit, hands an AI agent a description of the bug, and asks it to write a patch. The agent never sees the real upstream fix.
Every patch is then validated, audited, and graded in isolated containers, so you can see not just whether the model fixed the bug, but whether it introduced new ones along the way.
flowchart LR
spec[bug spec] --> clone
clone["clone<br/>(open net)"] --> generate
generate["generate<br/>(offline)"] --> validate
validate["validate<br/>(offline)"] --> ast["ast<br/>(offline)"]
generate -.->|"patch.diff"| store[(Postgres)]
validate -.->|"verdict"| store
ast -.->|"summary"| store
store --> ui[web UI + notebook]Each stage runs in its own container. Only clone has network access. Every
later stage is locked down to the LLM provider's API, so agents can't fetch
hints or the upstream fix mid-run.
| Feature | What it gives you |
|---|---|
| Repeated sampling | A run executes N iterations of the same input, so results are distributions, not anecdotes. |
| Campaigns | Sweep one bug across patcher variants and prompt styles, from a vague "fix this plz" to a full advisory, and compare outcomes on a live dashboard. |
| Outcome grading | Every patch lands in one of five scenarios, from S1 (clean fix) to S5 (didn't fix the bug and introduced a new vulnerability). |
| Cross-validation | Patches are re-judged by other models, and the headline numbers average the self- and cross-validation lenses so no single judge's bias dominates. |
| Cheat detection | An auditor flags iterations where the agent found the upstream fix instead of solving the bug itself. |
FLAWED drives the Claude, Codex, and Gemini CLIs head-to-head with the same inputs.
[!WARNING] FLAWED mounts the Docker socket (root-equivalent on the host) and executes untrusted third-party code inside its stage containers. Run it on a machine you trust to carry that workload. See
docs/security-model.md.
You need Docker, with the daemon socket accessible.
# 1. Configure. Writes .env for you (data dir + provider API keys)
./setup.sh
# 2. Bring up the stack (Postgres, API + worker, web UI, notebook)
docker compose up --build
# 3. Open http://127.0.0.1:8080
Then take your first run.
bugs/. Drag one
into the web UI's Bug Specs page, or use the CLI.
./scripts/import-all-bugs.sh
[!NOTE] The first run against a large upstream (e.g. Chromium) is slow. The validate stage clones the whole repo once to diff against the real upstream patch. The clone is cached and reused afterwards.
You need Docker, Node 20+, pnpm, uv, and
@devcontainers/cli (npm i -g @devcontainers/cli). Nix users can
nix-shell for everything except Docker.
make dev # uv sync + web deps
docker compose up -d postgres # FLAWED needs a Postgres to talk to
cp .env.example .env # points FLAWED_DB_URL at it
uv run flawed init # builds base images, creates the schema
uv run flawed serve # API + worker + webapp on port 8080
For the web dev loop, run make web-dev in a second terminal. It serves the
UI on port 5173 and proxies /api to flawed serve.
An isolated devcontainer for running AI coding agents against this repo safely
is documented in .devcontainer/README.md.
A bug spec is the unit of input. It carries a repo, a vulnerable commit, a bug description, an optional reproducer, and a set of prompt variants that model how the bug might realistically be reported (SAST finding, bug-bounty report, embargoed advisory, raw PoC, …). Specs are versioned and immutable, so a historical run's prompt and verdict never silently change.
The JSON contract lives in bugs.schema.json and is
documented in docs/bug-specs.md. All bundled specs
describe publicly disclosed, upstream-fixed vulnerabilities.
Postgres is the single source of truth. Run metadata and artifact bytes
(patches, verdicts, transcripts, logs) live in the database, so a deployment is
fully captured by its DB. The data/ directory is transient scratch that
stages mount at runtime.
scripts/export_dataset.py and
scripts/import_dataset.py (also available from the web UI).scripts/export_artifacts.py.uv run alembic upgrade head
runs automatically at startup).We publish pre-built datasets so you can load completed campaigns instead of
running everything yourself. Each dataset is a .tar.gz of one full snapshot,
hosted at https://flawed.s3.us-east-1.amazonaws.com.
All campaigns in a single bundle.
| Bundle | Archive |
|---|---|
| All campaigns | full.tar.gz |
| Campaign | Patcher model | Size | Archive |
|---|---|---|---|
| CVE-2026-22738 (Spring AI SpEL injection) | claude-opus-4-8 | 515.0 MB | .tar.gz |
| CVE-2026-22738 (Spring AI SpEL injection) | gpt-5.5 | 663.9 MB | .tar.gz |
CVE-2026-31431 (Linux algif_aead) | claude-opus-4-8 | 1.1 GB | .tar.gz |
CVE-2026-31431 (Linux algif_aead in-place) | gpt-5.5 | 966.9 MB | .tar.gz |
| CVE-2026-34197 (ActiveMQ Jolokia RCE) | claude-opus-4-8 | 645.2 MB | .tar.gz |
| CVE-2026-34197 (ActiveMQ Jolokia RCE) | gpt-5.5 | 819.0 MB | .tar.gz |
| CVE-2026-45185 (Exim BDAT use-after-free) | claude-opus-4-8 | 1.1 GB | .tar.gz |
| CVE-2026-45185 (Exim BDAT use-after-free) | gpt-5.5 | 1.0 GB |
Snapshots → Import datasets, paste one of the archive links above, and click
Find archives. FLAWED imports a direct .tar.gz link as-is and unpacks it
into your snapshots. Then restore it from the snapshots list (restore is a
separate, explicitly-confirmed step).
Import discovers multiple archives at once only when the URL is a listable bucket or a directory index page. Neither applies here, so import the links one at a time.
Download an archive, unpack it into the snapshots directory
(FLAWED_HOST_SNAPSHOT_DIR, or ./flawed-snapshots by default), and it shows
up in the snapshots list ready to restore.
curl -LO https://flawed.s3.us-east-1.amazonaws.com/compressed/campaigns/full/cve-2026-45185-exim-bdat-uaf_full_gpt-5.5_1782988292.tar.gz
mkdir -p "$FLAWED_HOST_SNAPSHOT_DIR"
tar -xzf cve-2026-45185-exim-bdat-uaf_full_gpt-5.5_1782988292.tar.gz -C "$FLAWED_HOST_SNAPSHOT_DIR"
Importing only stages the bundle on disk. Restoring it, which replaces the current deployment, is done separately from the snapshots page.
| Doc | What's in it |
|---|---|
docs/architecture.md | Components, data flow, storage layout |
docs/pipeline.md | The stages, iteration semantics, container layout |
docs/security-model.md | Per-stage network policy, sandboxing, threat model |
docs/bug-specs.md | Input contract, prompt composition, versioning |
docs/cli.md | flawed command reference + env vars |
docs/webapp.md | UI tour |
docs/notebook.md | JupyterLab integration + the flawed helper module |
packages/
flawed-core/ models, DB schema, settings
flawed-orchestrator/ run lifecycle, worker pool, devcontainer driver
flawed-stage-*/ the pipeline stages (clone, generate, validate, …)
flawed-agent-runner/ runs INSIDE the container (Claude / Codex / Gemini CLIs)
flawed-api/ FastAPI + SSE + notebook proxy
flawed-cli/ the `flawed` CLI
apps/web/ React + Vite + Tailwind webapp
containers/ base image, network lockdown, notebook container
bugs/ example bug specs
docs/ documentation (see above)
| Task | Command |
|---|---|
| Sync all Python + Node deps | make dev |
| Run tests | make test |
| Lint | make lint |
| Auto-format | make format |
| Web dev server (port 5173) | make web-dev |
| E2E smoke test | make e2e |
Tests need Docker. They spin up their own Postgres via testcontainers.
FLAWED is released under the MIT License. Contributions are welcome. See CONTRIBUTING.md, and please report security issues privately per SECURITY.md.
.tar.gz |
| CVE-2026-8512 (Chromium fsevents UAF) | claude-opus-4-8 | 500.7 MB | .tar.gz |
| CVE-2026-8512 (Chromium fsevents UAF) | gpt-5.5 | 342.7 MB | .tar.gz |
| freenginx-26 (Perl SV refcount UAF) | gpt-5.5 | 166.4 MB | .tar.gz |
| GHSA-wpqr-6v78-jr5g (Gemini CLI trust model) | claude-opus-4-8 | 1.2 GB | .tar.gz |
| GHSA-wpqr-6v78-jr5g (Gemini CLI trust model) | gpt-5.5 | 1.3 GB | .tar.gz |