Skip to content
KitploitKITPLOIT
ToolsExploitsBlog
Log in
Submit
ToolsExploitsBlog
Submit

Hacking, PenTest, and Cybersecurity Tools for Your Security Arsenal!

Kitploit is a directory of hacking, cybersecurity, and pentesting tools. Discover the latest project updates to find vulnerabilities, analyze systems, automate testing, and strengthen your security.

FeedsContactPrivacyΒ© 2026 Kitploit

Tool Directory

Categories

View all categories
Loading categories
dataset β€” Generative AI-based CyberSecurity-focused Prompt Dataset for Benchmarking Large Language Models | Kitploit
Tools/GitHubGitHub/cysecbench/dataset
IoT SecurityWeb Application ExploitationMalware AnalysisMachine LearningPapers & ResearchLearning & EducationCurated ResourcesAI Security
GitHubcysecbench/dataset

dataset

Generative AI-based CyberSecurity-focused Prompt Dataset for Benchmarking Large Language Models

View Repository
4510391 year agoReviewed by Kitploit
Share

πŸš€ CySecBench: Generative AI-based CyberSecurity-focused Prompt Dataset for Benchmarking Large Language Models πŸ›‘οΈ

Arxiv:CySecBench

License

The largest and most comprehensive Generative AI-based CyberSecurity-focused Dataset for Benchmarking Large Language Models

🌟 Overview

The CySecBench paper offers:

  • 🎯 A cutting-edge dataset of 12662 prompts tailored to cybersecurity challenges.
  • 🧠 Novel jailbreaking methods leveraging prompt obfuscation and refinement.
  • πŸ“Š Comprehensive performance evaluation of LLMs like ChatGPT, Claude, and Gemini.

Why CySecBench?

Existing datasets are too broad and often lack focus on cybersecurity. CySecBench fills this gap by providing domain-specific prompts organized into 10 categories, enabling a precise evaluation of LLM security mechanisms.

πŸ“„ Access the Paper

You can download the full research paper here: CySecBench (PDF)


✨ Features

πŸ—‚οΈ Dataset

  • πŸ“ 10 Categories of Prompts:
    • 🌩️ Cloud Attacks
    • βš™οΈ Control System Attacks
    • πŸ”’ Cryptographic Attacks
    • πŸ•΅οΈ Evasion Techniques
    • πŸ’» Hardware Attacks
    • πŸ” Intrusion Techniques
    • πŸ“‘ IoT Attacks
    • 🦠 Malware Attacks
    • 🌐 Network Attacks
    • 🌍 Web Application Attacks

πŸ—‚οΈ Repository Structure

/
β”œβ”€β”€ Code/
β”‚   β”œβ”€β”€ dataset_generation.py
β”‚   β”œβ”€β”€ keywords.txt
β”œβ”€β”€ Dataset/
β”‚   β”œβ”€β”€ Category sets/
β”‚   β”‚   β”œβ”€β”€ cysecbench-cloud-attacks.csv
β”‚   β”‚   β”œβ”€β”€ cysecbench-control-system-attacks.csv
β”‚   β”‚   β”œβ”€β”€ cysecbench-cryptographic-attacks.csv
β”‚   β”‚   β”œβ”€β”€ cysecbench-evasion-techniques.csv
β”‚   β”‚   β”œβ”€β”€ cysecbench-hardware-attacks.csv
β”‚   β”‚   β”œβ”€β”€ cysecbench-intrusion-techniques.csv
β”‚   β”‚   β”œβ”€β”€ cysecbench-iot-attacks.csv
β”‚   β”‚   β”œβ”€β”€ cysecbench-malware-attacks.csv
β”‚   β”‚   β”œβ”€β”€ cysecbench-network-attacks.csv
β”‚   β”‚   β”œβ”€β”€ cysecbench-web-application-attacks.csv
β”‚   β”œβ”€β”€ Full dataset/
β”‚   β”‚   β”œβ”€β”€ cysecbench.csv
β”‚   β”œβ”€β”€ Sample sets/
β”‚       β”œβ”€β”€ cysecbench-500.csv
β”‚       β”œβ”€β”€ cysecbench-2000.csv
β”‚       β”œβ”€β”€ cysecbench-6000.csv

πŸš€ Getting Started

βš™οΈ Prerequisites

  • 🐍 Python 3.8+
  • πŸ“¦ Required libraries: openai (only for dataset generation)

πŸ“Š Results using CySecBench

🎯 Evaluation Metrics

  • βœ… Success Rate (SR): Percentage of prompts bypassing ethical guidelines.
  • πŸ“ˆ Average Rating (AR): Degree of harmfulness in LLM responses (on a scale of 1-5, where 5 is the most harmful).

⚑ Jailbreaking Performance

LLMSuccess Rate (SR)Average Rating (AR)
πŸ€– Claude17.4%2.00
πŸ€– ChatGPT65.4%4.06
πŸ€– Gemini88.4%4.77

πŸ“œ Citation

If you use CySecBench, please cite:

@article{CySecBench2024,
	title        = {{CySecBench: Generative AI-based CyberSecurity-focused Prompt Dataset for Benchmarking Large Language Models}},
	author       = {Johan WahrΓ©us and Ahmed Mohamed Hussain and Panos Papadimitratos},
	year         = {2025},
	journal      = {arXiv preprint arXiv:2501.01335},
	url          = {https://arxiv.org/abs/2501.01335}
}

⭐ Star This Repository!

If you found CySecBench helpful or interesting, please give this repository a star ⭐ to show your support!


πŸ”’ License

This project is licensed under the MIT License. See the LICENSE file for details.

Download Tool

Most Popular

View all β†’

Discover the most used tools by our community.

No popularity data yet.

Explore all tools

Browse our collection of tools

View all tools β†’