Skip to content
KitploitKITPLOIT
ToolsExploitsBlog
Log in
Submit
ToolsExploitsBlog
Submit

Hacking, PenTest, and Cybersecurity Tools for Your Security Arsenal!

Kitploit is a directory of hacking, cybersecurity, and pentesting tools. Discover the latest project updates to find vulnerabilities, analyze systems, automate testing, and strengthen your security.

FeedsContactPrivacy© 2026 Kitploit

Tool Directory

Categories

View all categories
Loading categories
chatml-role-token-probe — Fire ChatML/Mistral/Llama control-token literals at your self-hosted agent stack and prove role-forgery before an attacker does (CVE-2026-61732). | Kitploit
Tools/GitHubGitHub/iceman2050/chatml-role-token-probe
Defensive ToolsVulnerability ScannersVulnerability AnalysisPenetration TestingDevSecOpsAI Security
GitHubiceman2050/chatml-role-token-probe

chatml-role-token-probe

Fire ChatML/Mistral/Llama control-token literals at your self-hosted agent stack and prove role-forgery before an attacker does (CVE-2026-61732).

Share
View RepositoryWebsite
11 month agoNot yet reviewed

chatml-role-token-probe

Prove your agent stack is vulnerable to role-forgery — before an attacker does.

A drop-in probe that fires a corpus of ChatML / Mistral / Llama control-token literals at your self-hosted agent stack's untrusted-text path and reports whether the tokenizer or server turns them into real control tokens — the primitive behind system-prompt forgery via prompt injection.

Why this exists (news hook)

On 2026-09-24, CVE-2026-61732 (CVSS 10.0) demonstrated the attack this tool detects: the same injected words, wrapped in ChatML control tokens, got executed by the model — while the plain-English version was refused. Hosted APIs strip these literals, but vLLM, SGLang, and Ollama do NOT strip them by default — so any self-hosted stack that concatenates untrusted text into prompts is forgeable until proven otherwise.

Source: Prompt Injection Your AI Guardrail Never Reads (AGTech Consulting, 2026-09-24).

Install

pip install .
# --tokenizer mode additionally needs: pip install tokenizers

Requires Python 3.10+. Standard library only for --selftest and --server.

Usage

# 1. Demonstration: two built-in mock tokenizers
#    (hardened stripper -> all PASS, naive passthrough -> all FORGEABLE)
chatml-role-token-probe probe --selftest

# 2. Probe a real Hugging Face tokenizer
chatml-role-token-probe probe --tokenizer meta-llama/Meta-Llama-3-8B-Instruct

# 3. Probe your running server's /tokenize endpoint
#    (POST {"text": ...} -> JSON {"ids": [...]})
chatml-role-token-probe probe --server http://localhost:8000/tokenize
chatml-role-token-probe probe --server http://localhost:8000/tokenize \
    --known-control-ids control-ids.json --json

Example output:

Probed 11 control-token literals: 11 FORGEABLE, 0 PASS/INCONCLUSIVE.
  [FORGEABLE] llama_bot_eot (llama3): marker '<|start_header_id|>' encodes to control token ids [128006] inside the literal.
  ...

Exit code is 1 when anything is FORGEABLE — drop it into CI to fail builds that regress token hygiene.

How verdicts work

For each corpus literal (e.g. <|im_start|>system) the probe compares:

ComparisonVerdict
literal tokenizes identically to its plain inner text (system)PASS — markers stripped/ignored
a marker's isolated token ids appear inside the literal's idsFORGEABLE — real control token
different from plain text, but no confirmed control-token matchINCONCLUSIVE — review manually

The corpus (src/roleprobe/corpus.json, 11 literals) covers ChatML (<|im_start|>, <|im_end|>), Mistral ([INST], <<SYS>>), Llama 3 (<|begin_of_text|>, <|start_header_id|>, <|eot_id|>), and bare role words (system:, assistant:).

Scope notes (honest)

  • --tokenizer only tests the tokenizer, not your serving stack's chat template or system-prompt handling — a PASS here does not prove your deployed endpoint is safe; use --server against the real path.
  • --server needs your endpoint to expose token ids; if it only returns text, this tool cannot confirm control-token status (INCONCLUSIVE).
  • Tokenizer behavior can differ between the tokenizers Rust port and a framework's Python port — probe the exact stack you ship.

Tests

PYTHONPATH=src pytest -q

Built by Ingenious AI

Part of the Ingenious AI platform suite — security tooling for the AI-agent era:

  • DisclosureGuard
  • ModelMinder
  • TermsWatch
  • Vibe-Code Audit
  • Rogue-Agent Defense
  • BreachHunter
  • ChatBreaker
  • Agent Flight Recorder
  • Agent Auth Gateway
  • DomainGuard
  • ContentCraft AI

License

MIT — see LICENSE.

Download Tool

Most Popular

View all →

Discover the most used tools by our community.

No popularity data yet.

Explore all tools

Browse our collection of tools

View all tools →