Skip to content
KitploitKITPLOIT
ToolsExploitsBlog
Log in
Submit
ToolsExploitsBlog
Submit

Hacking, PenTest, and Cybersecurity Tools for Your Security Arsenal!

Kitploit is a directory of hacking, cybersecurity, and pentesting tools. Discover the latest project updates to find vulnerabilities, analyze systems, automate testing, and strengthen your security.

FeedsContactPrivacy© 2026 Kitploit

Tool Directory

Categories

View all categories
Loading categories
cve-bench — A benchmark for evaluating AI agents on fixing real-world security vulnerabilities. | Kitploit
Tools/GitHubGitHub/giovannigatti/cve-bench
Static AnalysisDynamic Analysis (Sandboxing)Vulnerability AnalysisCode AnalysisPenetration TestingDevSecOpsMachine LearningLearning & EducationAI Security
GitHubgiovannigatti/cve-bench

cve-bench

A benchmark for evaluating AI agents on fixing real-world security vulnerabilities.

141765 months agoNot yet reviewed
View Repository
Share

CVE-Bench

Blog Harness Coverage

A benchmark for evaluating LLM agents on fixing real-world security vulnerabilities. Agents run inside sandboxed Docker containers and are scored against the maintainer's security test suite.


Requirements

  • Python 3.12+
  • Docker
  • OPENAI_API_KEY, ANTHROPIC_API_KEY, and/or POOLSIDE_API_KEY in your environment (or a .env file)

Install dependencies:

pip install poetry
poetry install

Task structure

Each task lives under tasks/{CVE-ID}/ and contains:

tasks/CVE-2026-33175/
├── meta.json           # GHSA ID, CWE, CVSS, repo URL, vulnerable and fixed SHAs
├── setup.sh            # Clones repo, checks out the vulnerable SHA, installs dependencies
├── run_tests.sh        # Injects test_security.py into the repo and runs pytest
├── test_security.py    # Security tests (xfail on vulnerable code, pass on the fix)
├── advisory.md         # Full GHSA advisory (richest prompt)
├── diagnose.md         # Behavioural description only — no file or function names
├── locate.md           # File and function only — no description of the flaw
└── Dockerfile          # Optional; only present when the task needs extra system deps

meta.json example:

{
  "ghsa_id": "GHSA-xxxx-xxxx-xxxx",
  "cwe": ["CWE-287"],
  "cvss": 9.1,
  "repo": {
    "url": "https://github.com/org/project",
    "vulnerable_sha": "abc123^",
    "fixed_sha": "abc123"
  }
}

setup.sh is idempotent and safe to re-run. test_security.py is kept hidden from the agent during the run and injected only after the agent finishes.


Building Docker images

python build.py

This builds:

  1. A shared base image (cve-bench/base) — Python 3.12, git, poetry, and the harness.
  2. One task image per task (cve-bench/{task-id}) — extends the base, copies the task directory, and runs setup.sh.

Options:

# Build specific tasks only
python build.py --task CVE-2026-33175 CVE-2026-42561

# Skip rebuilding the base image
python build.py --skip-base

Task images are built in parallel (up to 5 workers). If a task directory contains a Dockerfile, it is used instead of the generic docker/task.Dockerfile.


Validating tasks

Before running the benchmark, verify that each task's security tests correctly distinguish vulnerable from fixed code:

python validate.py

For each task, this runs three phases inside the task container:

PhaseWhat it checks
vulnerableSecurity tests must fail (or xfail) on the vulnerable SHA
fixedSecurity tests must pass on the fixed SHA
regressionNon-security tests must pass on the fixed SHA

Results are displayed as a live table. Exit code is 1 if any task fails any phase.

# Validate specific tasks only
python validate.py --task CVE-2026-33175 GHSA-r758-8hxw-4845

# Skip rebuilding images before validation
python validate.py --skip-build

Running the benchmark

python benchmark.py --model openai:gpt-5.5 poolside:laguna-m.1 --prompt-type advisory

Options:

FlagDescriptionDefault
--modelOne or more provider:model-id stringsall configured models
--prompt-typeadvisory, diagnose, locate, or any combinationall three
--taskOne or more task IDsall tasks
--cleanDelete existing results for the selected scope before startingoff

Supported providers:

ProviderFormatAPI key env var
OpenAIopenai:gpt-5.5OPENAI_API_KEY
Anthropicanthropic:claude-haiku-4-5-20251001ANTHROPIC_API_KEY
Poolsidepoolside:laguna-m.1POOLSIDE_API_KEY

Each run produces a JSON result file in results/:

results/{task-id}__{provider}:{model}__{prompt-type}.json

Existing result files are skipped automatically. Runs execute concurrently across tasks (up to 20 workers), with per-provider rate limiting (one active request per provider at a time) to avoid 429s.


Result format

Each result file is a JSON object with the following structure:

{
  "cve_id": "CVE-2026-33175",
  "model_id": "openai:gpt-5.5",
  "prompt_type": "advisory",
  "timestamp": "2026-05-01T12:00:00",
  "model_duration_s": 142.3,
  "test_duration_s": 8.1,
  "turns": [
    {
      "tool_calls_and_results": [...],
      "input_tokens": 12400,
      "output_tokens": 310
    }
  ],
  "tests": [
    {
      "kind": "security",
      "name": "test_email_verified",
      "outcome": "passed"
    }
  ]
}

tests[].kind is either "security" (from test_security.py) or "regression" (from the project's own test suite). A run is considered solved only if all security tests pass and no regression tests fail.


Generating charts

python generate_charts.py

Reads all result files from results/ and writes SVG charts to docs/images/charts/. Requires Chrome/Chromium for Bokeh's headless export (via chromedriver-binary).


Harness architecture

The harness runs inside each Docker container as python -m harness.run. It is responsible for loading the prompt, running the agentic loop, and writing the result file.

src/harness/
├── run.py                  # Entry point; parses args, wires components, calls BenchmarkRunner
├── client/
│   ├── factory.py          # Parses provider:model-id, returns the correct LLMClient
│   ├── _client.py          # Abstract LLMClient, ToolCall and LLMTurn dataclasses
│   ├── anthropic.py        # Anthropic SDK integration
│   └── oai.py              # OpenAI SDK integration (also used for Poolside)
├── agent/
│   ├── core.py             # Agentic loop: calls client, dispatches tool calls, threads messages
│   └── runner.py           # Wraps Agent, tracks timing and turn list
├── bench/
│   ├── runner.py           # Orchestrates setup → agent → security tests → regression tests
│   ├── result.py           # BenchmarkResult and TestResult dataclasses, JSON serialisation
│   └── repository.py       # Writes result files to disk
└── task/
    ├── tools.py             # Tool implementations: ListFiles, ReadFile, SearchInFiles,
    │                        #   EditFile, CreateFile, DeleteFile, RunPytest
    └── prompt_loader.py     # Reads advisory.md / diagnose.md / locate.md

Tools available to the agent:

ToolDescription
list_filesList files and directories in the repository
read_fileRead file contents, optionally a line range
search_in_filesRegex search across the codebase with optional file glob
edit_fileReplace a range of lines in an existing file
create_fileCreate a new file
delete_fileDelete a file
run_pytestRun the project's test suite; returns a JSON report

All tools validate paths against the repository root to prevent directory traversal. The agent does not have access to test_security.py or to the git history.

The agent loop runs for at most 20 turns. If the turn ceiling is reached, the run is recorded as-is and the security tests are still executed against whatever state the agent left the repository in.


Adding a task

  1. Create tasks/{CVE-ID}/ and add meta.json, setup.sh, run_tests.sh, test_security.py, advisory.md, diagnose.md, locate.md.
  2. Make setup.sh and run_tests.sh executable (chmod +x).
  3. Validate: python validate.py --task {CVE-ID}.
  4. Build: python build.py --task {CVE-ID}.

Disclosure

This work was conducted as independent research. At the time of conducting the research and preparing this repository, I had no institutional affiliation.


Citing this work

Download Tool

Most Popular

View all →

Discover the most used tools by our community.

  • 1

    OsintHunter

    letox4/osinthunter

    🎯 The ultimate OSINT toolkit: investigation graph, confidence scoring, alias families, map, timeline and reports. 49 modules, 400+ sites, no API keys required.

    786
  • 2

    ghostlock-app

    yukonga/ghostlock-app

    GhostLock One-Tap Execution App (CVE-2026-43499)

    506
  • 3

    ARTEX

    baarai/artex

    LLM multi-agent autonomous penetration testing platform with planner/worker agents, asset and exploration graphs, MITM traffic recording, and vulnerability retesting.

    429
  • 4

    redforge

    cybersecurity91/redforge

    Browser-side red team toolkit with 102 offline tools for recon, payload generation, credential attacks, crypto, encoding and engagement reporting.

    345
  • 5

    CyberStrikeAI

    ed1s0nz/cyberstrikeai

    The system of action for AI-native cybersecurity—where intent becomes governed execution, evidence becomes operational memory, and every operation improves the next.

    262
  • 6

    looking-glass

    anonghosty/looking-glass

    CSIRT platform that ingests Shadowserver reports into MongoDB and replays them as interactive maps, analysis, findings, advisories and executive dashboards.

    256

Explore all tools

Browse our collection of tools

View all tools →