CVE-Factory is a Multi-Agent system for fully automated, end-to-end CVE reproduction. Given CVE records, the system automatically researches details, generates test cases, builds Docker environments, and validates that each vulnerability can be both exploited and patched. The pipeline transforms CVE metadata into reproducible, testable vulnerability environments without manual intervention.

⚠️ Security Warning: This system builds and runs Docker containers containing vulnerable software. You MUST use the Docker-in-Docker (DinD) environment to isolate CVE containers from your host system. Never run CVE-Factory directly on your host Docker daemon.

📢 News

• [2026-03-27] Added 3,181 new CVE task environments (Hugging Face), Abacus-cve-v1.1 with 18.8k training traces, and LiveCVEBench-verified & PatchEval-verified benchmarks. Added 4 new agents (Judger, Changer, Comparer, Expert), 3 skills (cve-test-generator, cheat-detect, cheat-detect-evaluate), and switched tool access control from allowlist to denylist. See Update Notes for details.

✨ Highlights

🤖 End-to-End Automation

Input CVE records, get a complete CVE reproduction environment. Following the Terminal Bench standard, each generated task package includes:

• Environment Setup: Dockerfile and docker-compose.yaml hosting the vulnerable application
• Task Config: task.yaml containing structured instruction descriptions (CVE-identity-free)
• Reference Fix: solution.sh to patch the vulnerability
• Evaluation Entry: run-tests.sh to start the evaluation

Specifically designed for security tasks, our testing logic is split into:

• test_func.py: Functionality tests ensuring basic features work both before and after the fix
• test_vuln.py: Exploit tests verifying the vulnerability exists before patching and is resolved afterward

No manual research, no manual coding - fully automated from raw CVE metadata to validated reproduction.

Generated Artifact Structure:

CVE-2025-XXXX/
├── task.yaml              # Structured Task Metadata
├── Dockerfile          # Vulnerable Environment Setup
├── docker-compose.yaml # Service Orchestration   
├── task-deps/  
├── solution.sh            # Verified Patch
└── test/
    ├── test_func.py       # Functionality Check
    ├── test_vuln.py       # Vulnerability Exploit Check
    └── run-tests.sh           # One-click Evaluation Script 

📊 Proven High Success Rate

In a large-scale evaluation of 554 CVEs from 2025, CVE-Factory successfully reproduced 499 cases, achieving an 90.1% success rate. Furthermore, a rigorous expert review of 471 successful cases confirmed that 312 tasks (66.2%) were completely and accurately reproduced!

When compared against security experts using identical initial information, our system achieved a ~95% verification pass rate on environment and solution construction — demonstrating expert-level capability in automated vulnerability reproduction.

📂 Open Dataset: We release 1,000+ CVE task environments in the cve_tasks/ directory:

• trainset/ (887 tasks): Used for training Abacus-cve. The 4,000+ distilled agent traces on Hugging Face 🤗 are generated from these tasks using Claude Opus 4.5 with a Mini SWE-Agent harness.
• trainset-2/: Additional tasks with relatively simpler difficulty. Not included in the training data.
• NEW: Additional 3,181 tasks available at cve_tasks_3k_compressed on Hugging Face (compressed archive due to size limits), with 18.8k agent traces for training Abacus-cve-v1.1.

🚀 Training Results

Fine-tuning on CVE-Factory traces yields dramatic improvements across security benchmarks. Qwen3-32B achieves ~6.8× improvement on LiveCVEBench (5.29% → 35.79%), ~4.2× on PatchEval (5.66% → 23.58%), and even shows significant gains on Terminal-Bench (12.50% → 28.75%) — demonstrating strong cross-task generalization.

With just 4k traces, Abacus-cve (32B) outperforms Qwen3-Coder-480B, MiniMax-M2, and Claude Sonnet 4, approaching Claude Sonnet 4.5 level on security tasks.

NEW: Abacus-cve-v1.1 trained on 18.8k traces achieves further gains (+3.83 on LiveCVEBench, +2.38 on PatchEval). See cve_train_v1.1 for the expanded training data.

🧠 Autonomous Claude Code Agents

Unlike rigid retrieval workflows or simple tool-use loops, each agent operates as a full Claude Code session. We do not hard-code steps; instead, we define each agent by its Role (e.g., Analyzer), Goal (e.g., "Build a vulnerable environment"), Resources (e.g., Access to specific docs), and Verification Method (e.g., "Must pass check_env_ready"). Agents act like human developers: they autonomously explore files, debug errors, read logs, and iterate on solutions within their designated workspace.

⚡ Async Concurrent Processing

CVE-Factory is designed to handle multiple CVEs simultaneously. Each CVE pipeline executes asynchronously, meaning faster tasks proceed to subsequent stages without waiting for slower ones. The system uses an asynchronous architecture that allows you to separate concurrency limits for each specific Agent type. For example, you can set a higher limit for lightweight research tasks (Analyzer) and a lower limit for resource-intensive Docker tasks (Builder). This flexibility prevents system overload while maximizing processing speed. Stage-level timeouts ensure that hung processes don't block the processing queue.

🧩 Modular Multi-Stage Pipeline

The pipeline consists of 6 independent stages that can be run separately or combined.

• Phase 1 (Analyzer → Generator) performs CVE research and generates artifacts without requiring Docker.

  Tooling Requirement: The Analyzer agent relies on web_search and web_fetch tools. If you use a third-party API provider, you must ensure it supports these specific tool capabilities.

• Phase 2 (Builder → Validator → Solver → Checker) handles Docker environment construction and validation. From Environment Construction to Holistic Validation, no web-related tools are required, as the agents interact solely with the local filesystem and Docker daemon.

Each stage can also be invoked individually, enabling fine-grained control over the reproduction process and easy debugging of specific stages.

🏗️ Architecture

The system consists of 6 stages:

🚀 Quick Start

🐳 1. Set Up Docker-in-Docker Environment

# Start the isolated DinD environment (required for security)
cd dev-env
docker compose up -d

# Enter the development container
docker compose exec cve-factory bash

See dev-env/README.md for detailed DinD configuration and troubleshooting.

📂 2. Prepare CVE Input

Place the CVEs you want to reproduce in the original_cves_md/ directory. The files must be named in the format CVE-YYYY-NNNNN.md contained relevant information. We recommend using the cve-sampler from LiveCVEBench-Preview to prepare these inputs.

# Inside the DinD development container
cd /workspace
pip install -r requirements.txt

# Verify CVE input files are ready
ls original_cves_md/

▶️ 3. Run CVE-Factory

# Set API key or use Claude subscription
export ANTHROPIC_API_KEY="your-key"
export ANTHROPIC_BASE_URL="your-url"
# Process a specific CVE
python -m orchestrator.run --cve CVE-2025-XXXXX

# Or process all CVEs in the input directory
python -m orchestrator.run

# Run phases separately
python -m orchestrator.run --phase1  --cve CVE-2025-XXXXX # Analyzer + Generator only (no Docker needed)
python -m orchestrator.run --phase2  --cve CVE-2025-XXXXX # Builder → Checker (requires Docker)

A CVE reproduction is considered successful when:

• Vulnerable state: test_func.py PASS, test_vuln.py FAIL (app works, vulnerability exploitable)
• Fixed state: test_func.py PASS, test_vuln.py PASS (app works, vulnerability patched)

⚙️ Configuration

Key settings in config.yaml to optimize your run:

# Example config.yaml tweak
orchestrator:
  max_concurrent_cves: 3  # Lower concurrency for stability

agents:
  limits:
    builder: 2            # Prevent Docker from consuming all resources

📚 Documentation

• DinD Environment - Docker-in-Docker setup guide (start here)
• Scripts - Manual debugging and verification scripts
• Architecture - Detailed system design and data flow
• Agent Management - Orchestration and resource control
• Communication - Inter-agent message protocols
• Future Roadmap - Planned improvements and features

🚧 Ongoing Development

We are actively developing OneFactory, a Unified Synthetic Framework that integrates Terminal, SWE, and Security (CVE) capabilities into a comprehensive 3-in-1 agentic data pipeline.

Based on CVE-Factory, we have developed LiveCVEBench and released the first version of the benchmark, training data, and Abacus-cve model. We will continue to expand the benchmark and optimize our SFT & RL training recipes. Stay tuned for more updates!

🤝 Contributing

We are continuously expanding and updating this project. If you have any suggestions or would like to join/contribute to this project, please contact [email protected]!

📝 License

MIT License

🎓 Citation

@misc{luo2026cvefactory,
  title={CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability}, 
  author={Xianzhen Luo and Jingyuan Zhang and Shiqi Zhou and Rain Huang and Chuan Xiao and Qingfu Zhu and Zhiyuan Ma and Xing Yue and Yang Yue and Wencong Zeng and Wanxiang Che},
  year={2026},
  eprint={2602.03012},
  archivePrefix={arXiv},
  primaryClass={cs.CR},
  url={https://arxiv.org/abs/2602.03012}
}