
Generative AI-based CyberSecurity-focused Prompt Dataset for Benchmarking Large Language Models
The CySecBench paper offers:
Why CySecBench?
Existing datasets are too broad and often lack focus on cybersecurity. CySecBench fills this gap by providing domain-specific prompts organized into 10 categories, enabling a precise evaluation of LLM security mechanisms.
You can download the full research paper here: CySecBench (PDF)
/
βββ Code/
β βββ dataset_generation.py
β βββ keywords.txt
βββ Dataset/
β βββ Category sets/
β β βββ cysecbench-cloud-attacks.csv
β β βββ cysecbench-control-system-attacks.csv
β β βββ cysecbench-cryptographic-attacks.csv
β β βββ cysecbench-evasion-techniques.csv
β β βββ cysecbench-hardware-attacks.csv
β β βββ cysecbench-intrusion-techniques.csv
β β βββ cysecbench-iot-attacks.csv
β β βββ cysecbench-malware-attacks.csv
β β βββ cysecbench-network-attacks.csv
β β βββ cysecbench-web-application-attacks.csv
β βββ Full dataset/
β β βββ cysecbench.csv
β βββ Sample sets/
β βββ cysecbench-500.csv
β βββ cysecbench-2000.csv
β βββ cysecbench-6000.csv
openai (only for dataset generation)| LLM | Success Rate (SR) | Average Rating (AR) |
|---|---|---|
| π€ Claude | 17.4% | 2.00 |
| π€ ChatGPT | 65.4% | 4.06 |
| π€ Gemini | 88.4% | 4.77 |
If you use CySecBench, please cite:
@article{CySecBench2024,
title = {{CySecBench: Generative AI-based CyberSecurity-focused Prompt Dataset for Benchmarking Large Language Models}},
author = {Johan WahrΓ©us and Ahmed Mohamed Hussain and Panos Papadimitratos},
year = {2025},
journal = {arXiv preprint arXiv:2501.01335},
url = {https://arxiv.org/abs/2501.01335}
}
If you found CySecBench helpful or interesting, please give this repository a star β to show your support!
This project is licensed under the MIT License. See the LICENSE file for details.