
pytest for AI agents - Autonomous red-teaming, behavioral monitoring & security testing for LLM agents
ββββββββββββββ βββ βββ βββββββββββββββββ βββ ββββββββ βββββββββββββββββββ βββββββββββββββββββββββββ ββββββββ βββ βββββββββββ ββββββ ββββββββββββββ ββββββ βββ βββββββββββ ββββββ ββββββββββββββ ββββββ βββββββββββ βββββββββββββββββββββββββββββββββββββββββββββββ ββββββββββ βββ βββββββ βββββββββββββββββ ββββββββββββββββpytest for AI agents -- test, score, and harden before production
pip install crucible-security
π New to AI security? Read our Beginner's Getting Started Guide or set up a local test target with the n8n Local Demo Target Guide.
crucible init --target https://my-agent.com/api/chat
crucible scan --target https://my-agent.com/api/chat
crucible report crucible-report.json
One command. 90 attacks. Beautiful report.
crucible scan --output json pipes into any pipeline; fail builds on low gradesHow does Crucible compare to Garak and PyRIT? β See docs/comparison.md for a detailed, objective feature matrix.
What does Crucible test for? β See docs/owasp_mapping.md for the full OWASP Agentic AI Top 10 attack documentation (ASI01βASI10).
Need persistent dashboards, compliance reports, and team collaboration?
Join the waitlist for our upcoming cloud platform: crucible-cloud.vercel.app
| Module | Attacks | Status | OWASP Coverage |
|---|---|---|---|
| Prompt Injection | 50 | β Live | LLM01, LLM07 |
| Goal Hijacking | 20 | β Live | Agentic #1 |
| Jailbreaks | 20 | β Live | LLM01, LLM06 |
| Enterprise Graph | 10 | β Live | Agentic #2, #4 |
| Memory Poisoning | 8 | β Live | Agentic #5 |
| Infrastructure Escalation | 5 | β Live | LLM06, SSRF |
| Advanced Orchestration | 4 | β Live | Agentic #3 |
| MCP Security | 5 | β Live | Agentic #3 |
| MCP Server Scan | 10 | β Live (v0.4) | MCP-001 β MCP-005 |
| Behavioral Drift | multi-turn | β Live (v0.3) | Agentic #1, #2 |
| Multi-turn Attacks | strategies | β Live (v0.3) | LLM01, Agentic #1 |
| Deep Research Engine | autonomous | β Live (v0.4) | AI Research |
| Multi-Agent Contagion | orchestration | β Live (v0.4) | Agentic #2, #3 |
| Hallucination Detection | 15 | β Live (v0.5) | LLM09 / Agentic #9 |
| Toxicity & Content Safety | 20 | β Live (v0.5) | LLM01, LLM06 |
| Statistical Confidence | --confidence | β Live (v0.6) | Bootstrap & binomial bounds |
| MCP Trace Proxy | traffic proxy | β Live (v0.7) | Agentic #3 / Tool Misuse |
| Memory & RAG Poisoning | poison-test | β Live (v0.8) | Agentic #5 / Poisoning |
| Reference Targets | 12 targets | β Live (v0.18) | Ground-truth validation targets |
| # | Category | Crucible Module | Status |
|---|---|---|---|
| 1 | Goal Hijacking | goal_hijacking | Covered (20 attacks) |
| 2 | Prompt Injection | prompt_injection | Covered (50 attacks) |
| 3 | Tool Misuse | tool_injection / trace proxy | Covered (v0.7.0) |
| 4 | Identity Abuse | trace proxy + identity layer | Covered (v0.9.0) |
| 5 | Memory Poisoning | memory_poisoning / poison-test | Covered (8 attacks, v0.8.0) |
| 6 | Data Exfiltration | prompt_injection / exfiltration | Covered (v0.8.0) |
| 7 | Scope Violation | trace proxy | Covered (v0.7.0) |
| 8 | Cascading Failure | -- | Planned |
| 9 | Supply Chain / Overreliance | hallucination | Covered (15 attacks) |
| 10 | Rogue Agent | -- | Planned |
| Provider | Tested |
|---|---|
| OpenAI (GPT-4, GPT-4o) | Yes |
| Anthropic (Claude) | Yes |
| Groq (Llama, Mixtral) | Yes |
| Custom HTTP endpoint | Yes |
| LangChain (LangServe / FastAPI wrapper) | Yes |
| Ollama | Yes (v0.5) |
| LM Studio | Yes (v0.5) |
| HuggingFace TGI | Yes (v0.5) |
We provide several example scripts in the examples/ directory to help you get started:
| Script | Framework | Description |
|---|---|---|
test_openai_agent.py | OpenAI Chat Completions | Scan a raw OpenAI /chat/completions endpoint |
test_langchain_agent.py | LangChain (LangServe) | Scan a LangChain ReAct agent with OWASP LLM Top 10 mapping |
test_openai_assistant.py | OpenAI Assistants API | Scan an Assistants API wrapper endpoint |
All examples use respx to mock HTTP calls so they pass CI without a live server.
Running the LangChain Example:
python examples/test_langchain_agent.py
Running the OpenAI Assistant Example:
python examples/test_openai_assistant.py
Score starts at 100 and deducts per vulnerability found:
| Severity | Deduction |
|---|---|
| CRITICAL | -20 points |
| HIGH | -10 points |
| MEDIUM | -5 points |
| LOW | -2 points |
| Grade | Score Range |
|---|---|
| A | 90 -- 100 |
| B | 75 -- 89 |
| C | 60 -- 74 |
| D | 40 -- 59 |
| F | Below 40 |
# Generate config
crucible init --target URL --provider openai --key sk-xxx
# Run a standard scan
crucible scan \
--target https://my-agent.com/api/chat \
--name "My ChatBot" \
--header "Authorization: Bearer sk-xxx" \
--timeout 30 \
--concurrency 5
# Run with payload mutation (bypass WAFs/guardrails)
crucible scan --target URL --mutate
# Multi-turn attack strategy
crucible scan --target URL --strategy multi-turn
# Use agent profile to target attacks
crucible profile --target URL --output agent_profile.json
crucible scan --target URL --profile agent_profile.json
# Behavioral integrity audit (multi-turn drift detection)
crucible behavioral-audit \
--target https://my-agent.com/api/chat \
--baseline-turns 5 \
--probe-turns 15
# Generate EU AI Act compliance report from scan results
crucible scan --target URL --output json > results.json
crucible compliance-report --results results.json --output compliance.md
# JSON output for CI/CD
crucible scan --target URL --output json > report.json