
Open SAST benchmark corpus: 2,938,418 balanced cases, 11 languages, 21 framework targets, 242 emitted categories, and 231 distinct CWE IDs. Score SARIF 2.1.0 output with TPR, FPR, and Youden's J.
Ground truth for SAST. An open benchmark with machine-readable ground truth and deterministic SARIF scorers, for measuring how accurately a static analysis tool identifies true vulnerabilities in benchmark code and how often it flags safe code.
benchproctor.com · blog · Apache-2.0
Release status
Release 2026.11.16 is the current release. It is a curated release:
- 2,142 standalone cases in 11 languages across 21 framework targets, exactly 1,071 vulnerable and 1,071 safe;
- 60 multi-service applications in five languages, holding 860 labeled flows.
Releases resume in this curated form. Each release is a small, fresh draw: easy and medium difficulty only, on a fixed list of frameworks and weakness classes. The larger corpus it is drawn from is not published. See What this release does not contain.
Release 2026.07.22 is withdrawn. An audit found label and realism defects in families it was built from. Do not report scores against it. See Corrections and ground-truth disputes.
A SAST tool is only as trustworthy as its measured performance, and measuring that performance
requires ground truth. BenchProctor provides cases labeled vulnerable or safe, so you can score
compatible SARIF 2.1.0 output and calculate its true-positive rate, false-positive rate, and
Youden's J.
# 1. extract one language bundle
unzip Benchmarks/standalone/benchproctor-java-2026.11.16.zip -d benchproctor-java
# 2. scan one framework's testcode and export SARIF 2.1.0
your-tool scan ./benchproctor-java/spring/testcode --format sarif -o results.sarif
# 3. score against its answer key (standard-library Python, zero dependencies)
python ./benchproctor-java/score_sarif.py results.sarif \
./benchproctor-java/spring/expectedresults-2026.11.16.csv
# 4. read category-averaged and flat-aggregate TPR, FPR, and Youden's J
The scorer recovers each finding's CWE from the SARIF ruleId, the result or rule properties or
tags (for example external/cwe/cwe-089), or CWE taxa. This supports many common SARIF
encodings without configuration. If your tool emits no CWE, add --match-mode filename (any
finding on a vulnerable file counts; this rewards over-flagging, so prefer the default). The same
score_sarif.py ships inside every standalone bundle. The applications have their own scorer; see
Applications.
Many existing public SAST benchmarks exhibit one or more of these structural limitations:
sqli/Test01729_true_positive.java, a scanner can score well by matching the path rather than
analyzing the code.BenchProctor reduces exact-case memorization and label leakage by construction, covers 11 languages, and adds multi-service applications whose flows cross a process boundary.
| Release | 2026.11.16 |
| Standalone cases | 2,142: 1,071 vulnerable and 1,071 safe |
| Languages | 11: Java, Python, Go, Rust, TypeScript, JavaScript, PHP, Ruby, Bash, C, C++ |
| Framework targets | 21 (see the table below) |
| Weakness classes | 22, mapped to 22 distinct CWE IDs |
| Sampling | 3 vulnerable + 3 safe cases for every framework and class pair: 357 pairs |
| Applications | 60 two-service applications over 4 service graphs, in TypeScript, JavaScript, Python, Java and C++ |
| Application flows | 860 labeled flows: 234 vulnerable and 626 safe |
| Difficulty | easy and medium |
Every framework and class pair is balanced exactly. Under Youden's J, a flag-everything tool scores 0 because its true-positive and false-positive rates are both 100%. Some cases use shared runtime or companion files required by their language or framework; these files are not scored separately.
This is a curated release, and it is deliberately narrow. It does not contain:
Read a score on this release as a floor check on broadly known classes, not as a full assessment.
| Language | Framework targets | Targets | Classes per target | Cases |
|---|---|---|---|---|
| Java | Spring, Jakarta EE | 2 | 18 | 216 |
| Python | Flask, Django, FastAPI | 3 | 18 | 324 |
| Go | Gin, net/http | 2 | 15 | 180 |
| Rust | Actix-web, Axum | 2 | 15 | 180 |
| TypeScript | NestJS, Express | 2 | 18 | 216 |
| JavaScript | Express, Koa | 2 | 18 | 216 |
| PHP | Laravel, Symfony | 2 | 18 | 216 |
| Ruby | Rails, Sinatra | 2 | 17 | 204 |
| C++ | cpp-httplib, standalone | 2 | 19 | 228 |
| C | standalone | 1 | 16 | 96 |
| Bash | standalone | 1 | 11 | 66 |
| Total | 21 | 22 in union | 2,142 |
A framework target carries the classes that occur in its language. C and C++ include memory-safety classes; the managed languages do not.
| Class | CWE | Class | CWE |
|---|---|---|---|
| SQL injection | CWE-89 | Server-side request forgery | CWE-918 |
| OS command injection | CWE-78 | Code injection | CWE-94 |
| Path traversal | CWE-22 | Deserialization of untrusted data | CWE-502 |
| Cross-site scripting | CWE-79 | XML external entity | CWE-611 |
| LDAP injection | CWE-90 | Open redirect | CWE-601 |
| XPath injection | CWE-643 | Integer overflow | CWE-190 |
| Weak cipher | CWE-327 | NULL pointer dereference | CWE-476 |
| Weak hash | CWE-328 | Stack-based buffer overflow | CWE-121 |
| Weak random | CWE-330 | Heap-based buffer overflow | CWE-122 |
| Cookie without the secure attribute | CWE-614 | Buffer over-read | CWE-126 |
| Trust boundary violation | CWE-501 | Use after free | CWE-416 |