Skip to content
KitploitKITPLOIT
ツールエクスプロイトブログ
Log in
提出
ツールエクスプロイトブログ
提出

ハッキング、侵入テスト、サイバーセキュリティツールをあなたのセキュリティアーセナルに!

Kitploitはハッキング、サイバーセキュリティ、ペネトレーションテストのツールディレクトリです。最新のプロジェクトアップデートを見つけて、脆弱性の発見、システム分析、テストの自動化、セキュリティの強化を行いましょう。

フィードお問い合わせプライバシー© 2026 Kitploit

ツールディレクトリ

カテゴリ

すべてのカテゴリを見る
Loading categories
ツール/GitHubGitHub/tencent-hunyuan/diagsafe
Vulnerability AnalysisMachine LearningPapers & ResearchLearning & EducationRed TeamingCurated ResourcesAI SecurityLabs & Practice
GitHubtencent-hunyuan/diagsafe

DiagSafe

Diagnostic LLM safety benchmark with 3,489 bilingual prompts, an 84-label taxonomy, and prompt-level rubric judging for explainable, fine-grained model safety evaluation.

384645日前Kitploit レビュー済み
リポジトリを見る
ウェブサイト
共有
要求された言語のコンテンツは利用できません。英語版を表示しています。

English | 简体中文

Tencent HY

DiagSafe

Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics

License arXiv Dashboard Dataset


Overview

DiagSafe is a diagnostic safety benchmark for large language models, built from two parts:

  • A fine-grained, diverse safety benchmark: 3,489 prompts grounded in an 84-label, three-tier safety taxonomy, covering Chinese and English, single- and multi-turn dialogues, and direct harmful requests as well as several attack forms.
  • Prompt-level rubric evaluation: every prompt comes with its own safety items and their exemptions, combined with eight shared usefulness items; an LLM judge rules on each item and cites evidence.

Together they yield reliable, explainable judgments and fine-grained safety diagnoses of each model.

DiagSafe evaluation overview

(1) One rubric per prompt; (2) different judges reach consistent verdicts with the same rubric; (3) item-level hits expose the specific risks of each model.

Benchmark at a Glance

DimensionDetails
Prompts3,489 (DiagBase 2,401 · DiagAttack 1,088), Chinese and English at roughly 1:1
Taxonomy11 L1 / 34 L2 / 84 L3 labels, built from 204 safety provisions distilled from public AI safety guidelines of China, the US, the EU and other regions
Dialogues2,560 single-turn, 929 multi-turn (2–5 turns)
Attack formsDirect harmful requests, template jailbreaks, combined-template jailbreaks, instruction attacks
EvaluationLLM-as-a-judge: prompt-specific safety items (4.5 on average, each with exemptions) + eight shared usefulness items (0–3 score)

Every L3 label carries a definition, a general standard and detailed rules, and each rubric is generated from the standard of its label. The leaf-level comparison below is against SALAD-Bench, S-Eval, Do-Not-Answer and CHiSafetyBench: red leaves have no counterpart in those benchmarks, and orange leaves are covered there only by a parent category.

DiagSafe taxonomy comparison

DiagBase and DiagAttack

DiagBaseDiagAttack
Size2,4011,088
ConstructionSynthesized from the label standards along language × turns × attack formRewrites the single-turn direct requests of DiagBase into instruction attacks
Attack formsDirect harmful requests, template jailbreaks, combined-template jailbreaksInstruction attacks: the harmful intent is embedded in a benign-looking task with role, format and tone constraints
Dialogues1,472 single-turn, 929 multi-turnAll single-turn
FocusLabel coverage, safety across languages and multi-turn dialoguesThe conflict between strong instruction following and safety refusal

Prompt and rubric synthesis pipeline

Key Results

We evaluated 11 leading LLMs on DiagSafe. Two findings stand out.

1. Rubric-based evaluation is more reliable

Against 236 human-annotated responses as the gold standard:

ParadigmAgreement with humansCross-judge consistency (Cohen's κ)
Safety classifiers (MD-Judge, LlamaGuard4, S-Eval, Qwen3Guard)69.07% – 87.29%—
Generic safety prompt + LLM80.51% – 87.29%0.559 – 0.766
Label-specification prompt + LLM82.63% – 86.86%0.532 – 0.802
Prompt-level rubric + LLM (DiagSafe)89.41% – 95.34%0.828 – 0.864

Rubric-based judging reaches up to 95.34% agreement with human annotations and stays above 89% agreement across judge models, while the two prompt-based paradigms vary markedly between judges. Human review further finds 96.4% of prompts genuinely harmful and 99.6% of rubrics reasonable and actionable.

2. Leading LLMs remain unsafe, especially on DiagAttack

Unsafe rates on DiagBase and DiagAttack

  • On DiagBase, unsafe rates range from 2.8% to 32.9%.
  • On DiagAttack, every model's unsafe rate rises to 1.9–8.4× its DiagBase value (median 5.3×); 8 of the 11 models exceed 40%, and even the best model stays above 18%.
  • Wrapping a harmful intent in an ordinary task is enough to erode safety refusals: instruction following and safety refusal compete in current training, and existing benchmarks rarely cover this kind of attack.

Recently released models. We further evaluated five recently released models under exactly the same protocol (the same 3,489 prompts and prompt-level rubrics, no system prompt, the same decoding configuration, and Hy3 as the rubric-guided judge). Their results are consistent with the findings above: every model is clearly less safe on DiagAttack than on DiagBase.

Overall unsafe rate and usefulness of recently released models

Show exact numbers
ModelOverall ↓DiagBase ↓DiagAttack ↓Usefulness ↑
Qwen3.8-27B8.53.320.22.23
Hy4-Preview11.93.930.62.27
GLM-5.3†13.19.520.92.37
DeepSeek-V4.1-Flash18.55.847.52.14
Kimi-K326.814.953.52.31

Unsafe rates are in %; usefulness is the average score (0–3). † GLM-5.3 cannot disable its reasoning mode, so it is evaluated with the low reasoning-effort setting; all other settings are identical.

More diagnostics

Item-level judgments also expose differences that an overall unsafe rate hides. Left: each model on the safety–usefulness plane across the 11 L1 dimensions. Right: three models with nearly identical overall unsafe rates diverge sharply at the category level.

ツールをダウンロード

人気

すべて見る →

コミュニティで最も使われているツールを見つけましょう。

  • 1

    OsintHunter

    letox4/osinthunter

    🎯 究極のOSINTツールキット:調査グラフ、信頼度スコアリング、エイリアスファミリー、マップ、タイムライン、レポート。49個のモジュール、400以上のサイト、APIキー不要。

    654
  • 2

    ghostlock-app

    yukonga/ghostlock-app

    GhostLock ワンタップ実行アプリ(CVE-2026-43499)

    490
  • 3

    ARTEX

    baarai/artex

    プランナー/ワーカーエージェント、資産グラフと探索グラフ、MITMトラフィック記録、脆弱性の再テストを備えたLLMマルチエージェント自律ペネトレーションテストプラットフォーム。

    395
  • 4

    redforge

    cybersecurity91/redforge

    ブラウザ側のレッドチームツールキット。偵察、ペイロード生成、認証情報攻撃、暗号、エンコーディング、エンゲージメントレポート向けの102のオフラインツールを備えています。

    336
  • 5

    CyberStrikeAI

    ed1s0nz/cyberstrikeai

    AIネイティブなサイバーセキュリティのためのアクションシステム。意図がガバナンスされた実行に変わり、証拠が運用記憶となり、すべての操作が次の操作を改善する。

    258
  • 6

    antifox

    antifox-ai/antifox

    偵察、攻撃対象領域の探索、エクスプロイト検証、証拠に基づくレポート提供を実行するAIマルチエージェント自律ペネトレーションテストプラットフォーム。

    252

すべてのツールを探索

ツールコレクションを閲覧

すべてのツールを見る →