Skip to content
KitploitKITPLOIT
ИнструментыЭксплойтыБлог
Log in
Отправить
ИнструментыЭксплойтыБлог
Отправить

Инструменты для хакинга, пентеста и кибербезопасности — ваш арсенал защиты!

Kitploit — это каталог инструментов для хакинга, кибербезопасности и пентестинга. Находите последние обновления проектов для поиска уязвимостей, анализа систем, автоматизации тестирования и усиления вашей безопасности.

ЛентыКонтактыКонфиденциальность© 2026 Kitploit

Каталог инструментов

Категории

Все категории
Loading categories
Инструменты/GitHubGitHub/tencent-hunyuan/diagsafe
Vulnerability AnalysisMachine LearningPapers & ResearchLearning & EducationRed TeamingCurated ResourcesAI SecurityLabs & Practice
GitHubtencent-hunyuan/diagsafe

DiagSafe

Diagnostic LLM safety benchmark with 3,489 bilingual prompts, an 84-label taxonomy, and prompt-level rubric judging for explainable, fine-grained model safety evaluation.

384645 дней назадПроверено Kitploit
Репозиторий
Сайт
Поделиться
Контент недоступен на запрошенном языке. Показываем английскую версию.

English | 简体中文

Tencent HY

DiagSafe

Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics

License arXiv Dashboard Dataset


Overview

DiagSafe is a diagnostic safety benchmark for large language models, built from two parts:

  • A fine-grained, diverse safety benchmark: 3,489 prompts grounded in an 84-label, three-tier safety taxonomy, covering Chinese and English, single- and multi-turn dialogues, and direct harmful requests as well as several attack forms.
  • Prompt-level rubric evaluation: every prompt comes with its own safety items and their exemptions, combined with eight shared usefulness items; an LLM judge rules on each item and cites evidence.

Together they yield reliable, explainable judgments and fine-grained safety diagnoses of each model.

DiagSafe evaluation overview

(1) One rubric per prompt; (2) different judges reach consistent verdicts with the same rubric; (3) item-level hits expose the specific risks of each model.

Benchmark at a Glance

DimensionDetails
Prompts3,489 (DiagBase 2,401 · DiagAttack 1,088), Chinese and English at roughly 1:1
Taxonomy11 L1 / 34 L2 / 84 L3 labels, built from 204 safety provisions distilled from public AI safety guidelines of China, the US, the EU and other regions
Dialogues2,560 single-turn, 929 multi-turn (2–5 turns)
Attack formsDirect harmful requests, template jailbreaks, combined-template jailbreaks, instruction attacks
EvaluationLLM-as-a-judge: prompt-specific safety items (4.5 on average, each with exemptions) + eight shared usefulness items (0–3 score)

Every L3 label carries a definition, a general standard and detailed rules, and each rubric is generated from the standard of its label. The leaf-level comparison below is against SALAD-Bench, S-Eval, Do-Not-Answer and CHiSafetyBench: red leaves have no counterpart in those benchmarks, and orange leaves are covered there only by a parent category.

DiagSafe taxonomy comparison

DiagBase and DiagAttack

DiagBaseDiagAttack
Size2,4011,088
ConstructionSynthesized from the label standards along language × turns × attack formRewrites the single-turn direct requests of DiagBase into instruction attacks
Attack formsDirect harmful requests, template jailbreaks, combined-template jailbreaksInstruction attacks: the harmful intent is embedded in a benign-looking task with role, format and tone constraints
Dialogues1,472 single-turn, 929 multi-turnAll single-turn
FocusLabel coverage, safety across languages and multi-turn dialoguesThe conflict between strong instruction following and safety refusal

Prompt and rubric synthesis pipeline

Key Results

We evaluated 11 leading LLMs on DiagSafe. Two findings stand out.

1. Rubric-based evaluation is more reliable

Against 236 human-annotated responses as the gold standard:

ParadigmAgreement with humansCross-judge consistency (Cohen's κ)
Safety classifiers (MD-Judge, LlamaGuard4, S-Eval, Qwen3Guard)69.07% – 87.29%—
Generic safety prompt + LLM80.51% – 87.29%0.559 – 0.766
Label-specification prompt + LLM82.63% – 86.86%0.532 – 0.802
Prompt-level rubric + LLM (DiagSafe)89.41% – 95.34%0.828 – 0.864

Rubric-based judging reaches up to 95.34% agreement with human annotations and stays above 89% agreement across judge models, while the two prompt-based paradigms vary markedly between judges. Human review further finds 96.4% of prompts genuinely harmful and 99.6% of rubrics reasonable and actionable.

2. Leading LLMs remain unsafe, especially on DiagAttack

Unsafe rates on DiagBase and DiagAttack

  • On DiagBase, unsafe rates range from 2.8% to 32.9%.
  • On DiagAttack, every model's unsafe rate rises to 1.9–8.4× its DiagBase value (median 5.3×); 8 of the 11 models exceed 40%, and even the best model stays above 18%.
  • Wrapping a harmful intent in an ordinary task is enough to erode safety refusals: instruction following and safety refusal compete in current training, and existing benchmarks rarely cover this kind of attack.

Recently released models. We further evaluated five recently released models under exactly the same protocol (the same 3,489 prompts and prompt-level rubrics, no system prompt, the same decoding configuration, and Hy3 as the rubric-guided judge). Their results are consistent with the findings above: every model is clearly less safe on DiagAttack than on DiagBase.

Overall unsafe rate and usefulness of recently released models

Show exact numbers
ModelOverall ↓DiagBase ↓DiagAttack ↓Usefulness ↑
Qwen3.8-27B8.53.320.22.23
Hy4-Preview11.93.930.62.27
GLM-5.3†13.19.520.92.37
DeepSeek-V4.1-Flash18.55.847.52.14
Kimi-K326.814.953.52.31

Unsafe rates are in %; usefulness is the average score (0–3). † GLM-5.3 cannot disable its reasoning mode, so it is evaluated with the low reasoning-effort setting; all other settings are identical.

More diagnostics

Item-level judgments also expose differences that an overall unsafe rate hides. Left: each model on the safety–usefulness plane across the 11 L1 dimensions. Right: three models with nearly identical overall unsafe rates diverge sharply at the category level.

Скачать инструмент

Популярное

Смотреть все →

Откройте для себя самые используемые инструменты нашего сообщества.

  • 1

    OsintHunter

    letox4/osinthunter

    🎯 Полный набор инструментов OSINT: граф расследования, оценка достоверности, семейства псевдонимов, карта, временная шкала и отчёты. 49 модулей, более 400 сайтов, API-ключи не требуются.

    668
  • 2

    ghostlock-app

    yukonga/ghostlock-app

    # Приложение GhostLock с запуском в один клик (CVE-2026-43499)

    496
  • 3

    ARTEX

    baarai/artex

    Платформа автономного тестирования на проникновение на основе мультиагентной LLM с агентами-планировщиками и агентами-исполнителями, графами активов и исследования, записью MITM-трафика и повторной проверкой уязвимостей.

    394
  • 4

    redforge

    cybersecurity91/redforge

    Браузерный инструментарий для red team с 102 офлайн-инструментами для разведки, генерации полезной нагрузки, атак на учётные данные, криптографии, кодирования и отчётности по проектам.

    339
  • 5

    CyberStrikeAI

    ed1s0nz/cyberstrikeai

    Система действий для кибербезопасности на базе ИИ — где намерение превращается в управляемое исполнение, доказательства становятся оперативной памятью, а каждая операция улучшает следующую.

    258
  • 6

    antifox

    antifox-ai/antifox

    Мультиагентная платформа автономного тестирования на проникновение на базе ИИ, которая выполняет разведку, обнаружение поверхности атаки, валидацию эксплуатации и подготовку отчётов с подтверждающими доказательствами.

    254

Изучить все инструменты

Просмотрите нашу коллекцию инструментов

Смотреть все инструменты →