Extra sidecars per stack (Mailpit, MinIO, Redis, Jenkins, Prometheus, Ollama, …) are listed in The apps below.

The per-stack design docs live in benchmark-plans/ - start there for the full vulnerability catalog of each app. This README is the operational guide: how the repo is laid out and how to run and score an app.

Vulnerability classes covered

Every planted bug carries a CWE and an OWASP category. Rolled up by class - each bug counted once, under its primary CWE - the planted bugs break down roughly as (rollup last regenerated at 480 bugs; per-app counts above are current):

By OWASP category (2021 Top 10 for the web apps, API Top 10 2023 where the app is API-only):

Two non-web tracks sit alongside these: the network app plants 32 host/port and service-level findings for network scanners, and the two LLM apps (llmchat, llmagent) plant prompt-injection, tool-abuse and RAG-poisoning bugs scored on a separate injection-channel track.

Each bug is also tagged with a detection difficulty (118 E, 68 E-M, 202 M, 61 M-H, 100 H), a taint distance (351 in-file, 83 cross-file, 87 cross-service, 28 config) and a reachability (368 pre-auth, 181 user), so recall can be broken down along each of those axes instead of reported as one number. The per-app catalogs live in benchmark-plans/.

Repository structure

Running apps (dynast-bench CLI)

The CLI is the easiest way to drive the suite - it health-gates boots, arbitrates the shared ports, and speaks --json so a scanner harness can consume it. Needs Bun 1.2+ and Docker.

Full reference: dynast-bench/README.md.

Ports

Everything lives in a quiet slice of the ephemeral range, so the suite doesn't fight the usual 3000/8000/8080/5432 crowd - and every app owns a fixed port, so a URL always means the same app, alone or in a batch of five:

dynast-bench list is the map. If anything is already listening on a port an app owns, dynast-bench start leaves it alone and publishes that one service from the relocation pool instead, then prints (and --json-reports) the real URL. Nothing ever binds beyond 127.0.0.1. dynast-bench doctor shows which app ports are free; make targets don't relocate and publish the compose defaults (13311+) one app at a time, honouring DYNAST_PORT=<n>; --port N pins it.

Running apps (top-level Makefile)

The Makefiles remain the low-level contract and work standalone:

Two ways to run every app:

• Compose (make run) - the canonical multi-service topology the ground truth targets (app + Postgres/Redis/etc. as separate containers).
• Standalone (make solo) - one self-contained image per app (vuln/Dockerfile.standalone) with the datastores + an internal SSRF sink embedded, so docker build + docker run works with no compose. Behaviour and PoCs are identical (compose service names are aliased to 127.0.0.1).

The root stays deliberately small: this README, the design guide, the shared tooling, and the apps. Everything operational for a given app is inside that app's own folder.

Per-app anatomy

Every app under vulnerable-apps/ has the identical shape:

The vuln/safe twin

Each app ships two separated variant folders rather than git branches or patch files:

• vuln/ - the app with every planted bug. The default target; what a scanner points at.
• safe/ - the same app with every planted bug fixed and nothing else changed (parameterized queries, escaped output, added authz, safe deserializers, …).

diff -ru vulnerable-apps/<stack>/vuln vulnerable-apps/<stack>/safe is the ground truth. It must touch exactly the lines named in ground-truth/VULNERABILITIES.yaml and nothing else. Scanning the safe/ variant measures a tool's false-positive rate: every finding there is a false alarm, because the twin is clean by construction.

Because each variant's Docker build context is its own folder (vuln/ or safe/), the app's ground-truth/ sits outside every build context and cannot be baked into an image - the answer key can't leak into the running app, by construction.

Ground truth (ground-truth/)

Two answer keys, because there are two questions. VULNERABILITIES.yaml says what is wrong in the app; SURFACE.yaml says what is there at all.

VULNERABILITIES.yaml records one entry per planted bug:

SURFACE.yaml records one entry per operation the app exposes, vulnerable and benign alike - it is the denominator for endpoint coverage:

Benign operations are in there on purpose: a catalog of only the vulnerable routes would measure coverage of the answer key rather than of the app.

verify/ holds one runnable PoC per bug - it exits 0 against vuln/ and non-zero against safe/. That's the executable definition of "the bug is real (and actually fixed in the twin)."

The shared runner (dynast-bench/tools/poc-runner.sh) adds a third outcome the exit code cannot carry on its own: the harness could not run. Point the suite at a port nothing is listening on and a good half of any app's PoCs exit 1 - indistinguishable from a genuine fix. So the runner health-probes the target before it believes a rejection, applies a per-PoC deadline, and fails both legs on a timeout, a missing tool or a target that stopped answering. "The suite could not run" is never recorded as "the vulnerability is fixed".

Shared tooling (dynast-bench/, Bun/TypeScript)

One toolchain, used by every app, so cross-stack results are comparable:

• dynast-bench.ts - the CLI: start/stop/reset/clean, health gating, port arbitration, PoC verification, scoring, --json for harnesses.
• src/schema/ - types + validators for the two report formats (findings/v1, endpoints/v1) and the two answer keys (VULNERABILITIES.yaml, SURFACE.yaml), the CWE-family table used for partial credit, and the path/route/operation normalizers both sides of a comparison go through.
• src/normalize/ - adapters converting raw scanner output (OWASP ZAP, SARIF from Semgrep/CodeQL/Snyk, nuclei, Burp XML, nmap XML) into that format. The format is auto-detected, so score takes native output directly.
• src/scorer/ - matches findings against the answer key and emits precision / recall / F1, recall per difficulty / severity / reachability / taint / CWE, a discrimination score over the near-misses, and a duplicate (noise) ratio. Alongside it, an endpoint-coverage track grades how much of the app a run actually reached and splits every miss into "never found the endpoint" vs "found it, missed the bug".

examples/ holds files you can score immediately - a findings run, a false-positive run, three endpoint traces and two blank templates, each documented with the numbers it produces:

Full reference - the finding schema, the matching tiers, every metric: dynast-bench/README.md.

Uniform Makefile interface (identical in every app)

The apps

Plus three API-only apps (GraphQL, WebSocket, Swagger/OpenAPI), a network-range fleet for host/port scanners, and two LLM apps:

Both LLM apps run a local model via an internal-only Ollama container (gemma3:1b for chat, qwen2.5:1.5b for tool calling) - no API key, no egress, no per-run cost - and ship a scripted LLM_BACKEND=stub backend so the ground-truth PoCs stay deterministic against a stochastic model.

See benchmark-plans/README.md for the shared domain model, the OWASP-Top-10 coverage matrix, and the benchmark-design principles (near-misses, taint distance, logic-only bugs).

Getting started

Or drive one app directly with its Makefile:

Status

• 19 apps carry a complete answer key: 549 planted vulnerabilities, 146 near-misses, 546 PoCs, and a Dockerfile.standalone each (--solo). dynast-bench list prints the live table.
• nextjs is the reference implementation - built and validated end to end (35 vulns + 15 near-misses). make validate APP=nextjs proves every PoC exploitable on vuln/ and fixed on safe/; make solo APP=nextjs runs it from one image. Copy its patterns.
• dynast-bench CLI - built: runs, verifies, scores and cleans any app, in compose or single-image mode, with --json for harnesses.
• The scorer - built (dynast-bench/src/): scanner output → normalized findings → precision/recall/F1, per-difficulty recall, a discrimination score over the near-misses, and separate discovery (network) and injection-channel (LLM) tracks.
• Endpoint coverage - built: a SURFACE.yaml per app (~600 operations across the fleet) grading how much of the app a run actually reached, and splitting every miss into "never found the endpoint" vs "found it, missed the bug".
• Per-app invariants over all 19 answer keys and surface catalogs run in make test; dynast-bench check --all is the CI gate.

Scoring a tool

score reads a findings/v1 file or native ZAP / SARIF / nuclei / Burp / nmap output - the format is sniffed. Start from examples/ if you are wiring up a tool: examples/template-findings.json is a blank skeleton with every field, and examples/findings.json is a working file you can score right now.

Endpoint discovery is graded separately, against each app's SURFACE.yaml:

That is what separates a discovery miss (never reached the endpoint - fix the crawler) from an analysis miss (reached it, did not report - fix the scanner). See dynast-bench/README.md for the schema, the matching tiers and every metric.

Reading the report (Leg │ Precision │ Recall │ F1)

A leg is one scan run against one target state:

All three run 0.0–1.0, and for all three higher is better (1.0 is perfect):

The one inversion: on the safe-twin leg there is nothing real to find, so every finding there is a false alarm - fewer is better, and an empty report is the perfect score.

Endpoint coverage

Recall tells you how many bugs a tool found. It cannot tell you why it missed the rest - and the two reasons need opposite fixes:

Telling them apart needs a second input: the endpoints your tool says it found. That is endpoints/v1, scored against each app's SURFACE.yaml.

A crawler that reads HTML and runs JS but never completes a multi-step flow:

The tier breakdown is the useful part: 100% on static-html and 0% on flow is a discovery problem, not a scanner problem, and those read identically in a single recall number.

Two rules keep the number honest:

• Transport is not operation. One POST /graphql does not exercise the 25 GraphQL operations behind it; one WebSocket handshake does not exercise its events; one POST /api/runs does not exercise an agent's tools. Reaching a URL and exercising what lives there are separately scored.
• Missing telemetry produces no track at all, never 0%. "We did not measure this" and "it reached nothing" are opposite claims about a tool.

Reported endpoints that match nothing cost precision but never reduce coverage, so spraying a wordlist is not a way to score higher. Full model: dynast-bench/README.md#endpoint-coverage.

License

dynast-bench is made with ♥ by @j3ssie to benchmark Vigolium and Gimorra (An autonomous offensive-security agent), and it is released under the MIT license.