Skip to content
KitploitKITPLOIT
ToolsExploitsBlog
Log in
Submit
ToolsExploitsBlog
Submit

Hacking, PenTest, and Cybersecurity Tools for Your Security Arsenal!

Kitploit is a directory of hacking, cybersecurity, and pentesting tools. Discover the latest project updates to find vulnerabilities, analyze systems, automate testing, and strengthen your security.

FeedsContactPrivacy© 2026 Kitploit

Tool Directory

Categories

View all categories
Loading categories
BenchProctor — Open SAST benchmark corpus: 2,938,418 balanced cases, 11 languages, 21 framework targets, 242 emitted categories, and 231 distinct CWE IDs. Score SARIF 2.1.0 output with TPR, FPR, and Youden's J. | Kitploit
Tools/GitHubGitHub/theauditortool/benchproctor
Static AnalysisWeb Vulnerability ScannersVulnerability AnalysisCode AnalysisLearning & Education
GitHubtheauditortool/benchproctor

BenchProctor

Open SAST benchmark corpus: 2,938,418 balanced cases, 11 languages, 21 framework targets, 242 emitted categories, and 231 distinct CWE IDs. Score SARIF 2.1.0 output with TPR, FPR, and Youden's J.

View RepositoryWebsite
1331 day agoNot yet reviewed
Share

BenchProctor

Ground truth for SAST. An open benchmark with machine-readable ground truth and deterministic SARIF scorers, for measuring how accurately a static analysis tool identifies true vulnerabilities in benchmark code and how often it flags safe code.

benchproctor.com · blog · Apache-2.0

Release status

Release 2026.11.16 is the current release. It is a curated release:

  • 2,142 standalone cases in 11 languages across 21 framework targets, exactly 1,071 vulnerable and 1,071 safe;
  • 60 multi-service applications in five languages, holding 860 labeled flows.

Releases resume in this curated form. Each release is a small, fresh draw: easy and medium difficulty only, on a fixed list of frameworks and weakness classes. The larger corpus it is drawn from is not published. See What this release does not contain.

Release 2026.07.22 is withdrawn. An audit found label and realism defects in families it was built from. Do not report scores against it. See Corrections and ground-truth disputes.

A SAST tool is only as trustworthy as its measured performance, and measuring that performance requires ground truth. BenchProctor provides cases labeled vulnerable or safe, so you can score compatible SARIF 2.1.0 output and calculate its true-positive rate, false-positive rate, and Youden's J.

Quick start

# 1. extract one language bundle
unzip Benchmarks/standalone/benchproctor-java-2026.11.16.zip -d benchproctor-java

# 2. scan one framework's testcode and export SARIF 2.1.0
your-tool scan ./benchproctor-java/spring/testcode --format sarif -o results.sarif

# 3. score against its answer key (standard-library Python, zero dependencies)
python ./benchproctor-java/score_sarif.py results.sarif \
  ./benchproctor-java/spring/expectedresults-2026.11.16.csv

# 4. read category-averaged and flat-aggregate TPR, FPR, and Youden's J

The scorer recovers each finding's CWE from the SARIF ruleId, the result or rule properties or tags (for example external/cwe/cwe-089), or CWE taxa. This supports many common SARIF encodings without configuration. If your tool emits no CWE, add --match-mode filename (any finding on a vulnerable file counts; this rewards over-flagging, so prefer the default). The same score_sarif.py ships inside every standalone bundle. The applications have their own scorer; see Applications.

Why another benchmark

Many existing public SAST benchmarks exhibit one or more of these structural limitations:

  • Fixed and unchanging cases. A static set of human-written cases can encourage benchmark-specific tuning over time, making results less indicative of generalization to unseen code.
  • Labels encoded in filenames. When a test lives at sqli/Test01729_true_positive.java, a scanner can score well by matching the path rather than analyzing the code.
  • Limited structural coverage. Many suites focus on one language and single-file findings, without testing a flow that crosses a service boundary.

BenchProctor reduces exact-case memorization and label leakage by construction, covers 11 languages, and adds multi-service applications whose flows cross a process boundary.

What's in this release

Release2026.11.16
Standalone cases2,142: 1,071 vulnerable and 1,071 safe
Languages11: Java, Python, Go, Rust, TypeScript, JavaScript, PHP, Ruby, Bash, C, C++
Framework targets21 (see the table below)
Weakness classes22, mapped to 22 distinct CWE IDs
Sampling3 vulnerable + 3 safe cases for every framework and class pair: 357 pairs
Applications60 two-service applications over 4 service graphs, in TypeScript, JavaScript, Python, Java and C++
Application flows860 labeled flows: 234 vulnerable and 626 safe
Difficultyeasy and medium

Every framework and class pair is balanced exactly. Under Youden's J, a flag-everything tool scores 0 because its true-positive and false-positive rates are both 100%. Some cases use shared runtime or companion files required by their language or framework; these files are not scored separately.

What this release does not contain

This is a curated release, and it is deliberately narrow. It does not contain:

  • Cases built around an ineffective safeguard. In this release a vulnerable case has no safeguard, and a safe case has one that works. A tool that treats the presence of any safeguard as proof of safety is not penalized here.
  • The deepest propagation patterns. Flows use direct and medium-depth propagation only.
  • Frameworks, weakness classes and application shapes beyond the lists on this page.

Read a score on this release as a floor check on broadly known classes, not as a full assessment.

How it's built

  • Combinatorial, not hand-written. Each class models security-relevant behavior using the applicable components of four axes: where untrusted input enters (source), how it travels (propagator), what would neutralize it (sanitizer or safeguard), and the dangerous operation it reaches (sink). A vulnerable case omits the safeguard; its safe counterpart applies an effective one. Every emitted case is constrained to a valid, analyzable path using real language and framework APIs.
  • Synthetic by design. The corpus intentionally isolates scoring-relevant security properties. Individual cases use real APIs but are not intended to represent complete production features.
  • Anti-leakage by construction. Emitted files carry no comments, no CWE tags, no category names, or label-bearing identifiers. File IDs are shuffled, so filenames do not encode a case's class or label. The CSV answer keys are the only published source of labels.
  • Seed rotation. Each release is drawn with its own seed. A new release keeps the same languages, frameworks, classes, sampling and difficulty, and emits different combinations and identifiers. This limits the usefulness of exact-file and filename memorization and is designed to support cross-release score comparability. Structural generalization across releases remains possible and is expected.

Languages & frameworks

LanguageFramework targetsTargetsClasses per targetCases
JavaSpring, Jakarta EE218216
PythonFlask, Django, FastAPI318324
GoGin, net/http215180
RustActix-web, Axum215180
TypeScriptNestJS, Express218216
JavaScriptExpress, Koa218216
PHPLaravel, Symfony218216
RubyRails, Sinatra217204
C++cpp-httplib, standalone219228
Cstandalone11696
Bashstandalone11166
Total2122 in union2,142

A framework target carries the classes that occur in its language. C and C++ include memory-safety classes; the managed languages do not.

Weakness classes

ClassCWEClassCWE
SQL injectionCWE-89Server-side request forgeryCWE-918
OS command injectionCWE-78Code injectionCWE-94
Path traversalCWE-22Deserialization of untrusted dataCWE-502
Cross-site scriptingCWE-79XML external entityCWE-611
LDAP injectionCWE-90Open redirectCWE-601
XPath injectionCWE-643Integer overflowCWE-190
Weak cipherCWE-327NULL pointer dereferenceCWE-476
Weak hashCWE-328Stack-based buffer overflowCWE-121
Weak randomCWE-330Heap-based buffer overflowCWE-122
Cookie without the secure attributeCWE-614Buffer over-readCWE-126
Trust boundary violationCWE-501Use after freeCWE-416
Download Tool

Most Popular

View all →

Discover the most used tools by our community.

  • 1

    OsintHunter

    letox4/osinthunter

    🎯 The ultimate OSINT toolkit: investigation graph, confidence scoring, alias families, map, timeline and reports. 49 modules, 400+ sites, no API keys required.

    491
  • 2

    ghostlock-app

    yukonga/ghostlock-app

    GhostLock One-Tap Execution App (CVE-2026-43499)

    462
  • 3

    opencti

    opencti-platform/opencti

    Open Cyber Threat Intelligence Platform

    345
  • 4

    ARTEX

    baarai/artex

    LLM multi-agent autonomous penetration testing platform with planner/worker agents, asset and exploration graphs, MITM traffic recording, and vulnerability retesting.

    340
  • 5

    redforge

    cybersecurity91/redforge

    Browser-side red team toolkit with 102 offline tools for recon, payload generation, credential attacks, crypto, encoding and engagement reporting.

    306
  • 6

    estorides

    grisuno/estorides

    Aggregates 99+ free OSINT sources in parallel, resolves entities into a knowledge graph, and exports STIX 2.1, MISP, or GraphML with LLM-assisted analysis.

    292

Explore all tools

Browse our collection of tools

View all tools →