crucible

   ██████╗██████╗ ██╗   ██╗ ██████╗██╗██████╗ ██╗     ███████╗
  ██╔════╝██╔══██╗██║   ██║██╔════╝██║██╔══██╗██║     ██╔════╝
  ██║     ██████╔╝██║   ██║██║     ██║██████╔╝██║     █████╗
  ██║     ██╔══██╗██║   ██║██║     ██║██╔══██╗██║     ██╔══╝
  ╚██████╗██║  ██║╚██████╔╝╚██████╗██║██████╔╝███████╗███████╗
   ╚═════╝╚═╝  ╚═╝ ╚═════╝  ╚═════╝╚═╝╚═════╝ ╚══════╝╚══════╝
  

pytest for AI agents -- test, score, and harden before production

---

Install

pip install crucible-security

Quick Start

🆕 New to AI security? Read our Beginner's Getting Started Guide or set up a local test target with the n8n Local Demo Target Guide.

crucible init --target https://my-agent.com/api/chat
crucible scan --target https://my-agent.com/api/chat
crucible report crucible-report.json

One command. 90 attacks. Beautiful report.

Why Crucible?

• Behavioral integrity testing -- the only tool that tests agent behavior across conversations, not just single-shot attacks
• Automated red-teaming -- 90+ real attack payloads run in under 60 seconds, not weeks of manual testing
• OWASP-aligned -- maps every attack to the OWASP Top 10 for LLM Applications and OWASP Agentic Top 10
• CI/CD native -- crucible scan --output json pipes into any pipeline; fail builds on low grades
• Regulatory compliance -- auto-generate EU AI Act 2024 compliance reports from scan results
• MCP security -- the only tool with a native Model Context Protocol security module

How does Crucible compare to Garak and PyRIT? → See docs/comparison.md for a detailed, objective feature matrix.

What does Crucible test for? → See docs/owasp_mapping.md for the full OWASP Agentic AI Top 10 attack documentation (ASI01–ASI10).

☁️ Crucible Cloud (Waitlist)

Need persistent dashboards, compliance reports, and team collaboration?
Join the waitlist for our upcoming cloud platform: crucible-cloud.vercel.app

Modules

Module	Attacks	Status	OWASP Coverage
Prompt Injection	50	✅ Live	LLM01, LLM07
Goal Hijacking	20	✅ Live	Agentic #1
Jailbreaks	20	✅ Live	LLM01, LLM06
Enterprise Graph	10	✅ Live	Agentic #2, #4
Memory Poisoning	8	✅ Live	Agentic #5
Infrastructure Escalation	5	✅ Live	LLM06, SSRF
Advanced Orchestration	4	✅ Live	Agentic #3
MCP Security	5	✅ Live	Agentic #3
MCP Server Scan	10	✅ Live (v0.4)	MCP-001 – MCP-005
Behavioral Drift	multi-turn	✅ Live (v0.3)	Agentic #1, #2
Multi-turn Attacks	strategies	✅ Live (v0.3)	LLM01, Agentic #1
Deep Research Engine	autonomous	✅ Live (v0.4)	AI Research
Multi-Agent Contagion	orchestration	✅ Live (v0.4)	Agentic #2, #3
Hallucination Detection	15	✅ Live (v0.5)	LLM09 / Agentic #9
Toxicity & Content Safety	20	✅ Live (v0.5)	LLM01, LLM06
Statistical Confidence	--confidence	✅ Live (v0.6)	Bootstrap & binomial bounds
MCP Trace Proxy	traffic proxy	✅ Live (v0.7)	Agentic #3 / Tool Misuse
Memory & RAG Poisoning	poison-test	✅ Live (v0.8)	Agentic #5 / Poisoning
Reference Targets	12 targets	✅ Live (v0.18)	Ground-truth validation targets

OWASP Agentic Top 10 Coverage

#	Category	Crucible Module	Status
1	Goal Hijacking	goal_hijacking	Covered (20 attacks)
2	Prompt Injection	prompt_injection	Covered (50 attacks)
3	Tool Misuse	tool_injection / trace proxy	Covered (v0.7.0)
4	Identity Abuse	trace proxy + identity layer	Covered (v0.9.0)
5	Memory Poisoning	memory_poisoning / poison-test	Covered (8 attacks, v0.8.0)
6	Data Exfiltration	prompt_injection / exfiltration	Covered (v0.8.0)
7	Scope Violation	trace proxy	Covered (v0.7.0)
8	Cascading Failure	--	Planned
9	Supply Chain / Overreliance	hallucination	Covered (15 attacks)
10	Rogue Agent	--	Planned

Supported Providers

Provider	Tested
OpenAI (GPT-4, GPT-4o)	Yes
Anthropic (Claude)	Yes
Groq (Llama, Mixtral)	Yes
Custom HTTP endpoint	Yes
LangChain (LangServe / FastAPI wrapper)	Yes
Ollama	Yes (v0.5)
LM Studio	Yes (v0.5)
HuggingFace TGI	Yes (v0.5)

Examples

We provide several example scripts in the examples/ directory to help you get started:

Script	Framework	Description
test_openai_agent.py	OpenAI Chat Completions	Scan a raw OpenAI /chat/completions endpoint
test_langchain_agent.py	LangChain (LangServe)	Scan a LangChain ReAct agent with OWASP LLM Top 10 mapping
test_openai_assistant.py	OpenAI Assistants API	Scan an Assistants API wrapper endpoint

All examples use respx to mock HTTP calls so they pass CI without a live server.

Running the LangChain Example:

python examples/test_langchain_agent.py

Running the OpenAI Assistant Example:

python examples/test_openai_assistant.py

Scoring System

Score starts at 100 and deducts per vulnerability found:

Severity	Deduction
CRITICAL	-20 points
HIGH	-10 points
MEDIUM	-5 points
LOW	-2 points

Grade	Score Range
A	90 -- 100
B	75 -- 89
C	60 -- 74
D	40 -- 59
F	Below 40

CLI Reference

# Generate config
crucible init --target URL --provider openai --key sk-xxx

# Run a standard scan
crucible scan \
  --target https://my-agent.com/api/chat \
  --name "My ChatBot" \
  --header "Authorization: Bearer sk-xxx" \
  --timeout 30 \
  --concurrency 5

# Run with payload mutation (bypass WAFs/guardrails)
crucible scan --target URL --mutate

# Multi-turn attack strategy
crucible scan --target URL --strategy multi-turn

# Use agent profile to target attacks
crucible profile --target URL --output agent_profile.json
crucible scan --target URL --profile agent_profile.json

# Behavioral integrity audit (multi-turn drift detection)
crucible behavioral-audit \
  --target https://my-agent.com/api/chat \
  --baseline-turns 5 \
  --probe-turns 15

# Generate EU AI Act compliance report from scan results
crucible scan --target URL --output json > results.json
crucible compliance-report --results results.json --output compliance.md

# JSON output for CI/CD
crucible scan --target URL --output json > report.json