praetorian-inc/augustusLLM security testing framework for detecting prompt injection, jailbreaks, and adversarial attacks — 190+ probes, 28 providers, single Go binary
Augustus - LLM vulnerability scanner for prompt injection, jailbreak, and adversarial attack testing
# Augustus - LLM Vulnerability Scanner
> Test large language models against 210+ adversarial attacks covering prompt injection, jailbreaks, encoding exploits, and data extraction.
[](https://github.com/praetorian-inc/augustus/actions/workflows/ci.yaml)
[](go.mod)
[](LICENSE)
[](https://goreportcard.com/report/github.com/praetorian-inc/augustus)
[](https://github.com/praetorian-inc/augustus/releases)
**Augustus** is a Go-based LLM vulnerability scanner for security professionals. It tests large language models against a wide range of adversarial attacks, integrates with 28 LLM providers, and produces actionable vulnerability reports.
Unlike research-oriented tools, Augustus is built for production security testing — concurrent scanning, rate limiting, retry logic, and timeout handling come out of the box.
## Table of Contents
- [Why Augustus](#why-augustus)
- [Features](#features)
- [Quick Start](#quick-start)
- [Supported Providers](#supported-providers)
- [Usage](#usage)
- [Single Probe](#single-probe)
- [Multiple Probes](#multiple-probes)
- [Buff Transformations](#buff-transformations)
- [Output Formats](#output-formats)
- [Custom REST Endpoints](#custom-rest-endpoints)
- [How It Works](#how-it-works)
- [Architecture](#architecture)
- [Configuration](#configuration)
- [FAQ](#faq)
- [Troubleshooting](#troubleshooting)
- [Contributing](#contributing)
- [Security](#security)
- [Support](#support)
- [License](#license)
## Why Augustus
| Feature | Augustus | garak | promptfoo |
|---------|----------|-------|-----------|
| Language | Go | Python | TypeScript |
| Single binary | Yes | No | No |
| Concurrent scanning | Goroutine pools | Multiprocessing pools | Yes |
| LLM providers | 28 | 35+ | 80+ |
| Probe types | 210+ | 160+ | 119 plugins + 36 strategies |
| Enterprise focus | Yes | Research | Yes |
## Features
| Feature | Description |
|---------|-------------|
| **210+ Vulnerability Probes** | 47 attack categories: jailbreaks, prompt injection, adversarial examples, data extraction, safety benchmarks, agent attacks, and more |
| **28 LLM Providers** | OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Ollama, and 22 more with 43 generator variants |
| **90+ Detectors** | Pattern matching, LLM-as-a-judge, HarmJudge (arXiv:2511.15304), Perspective API, unsafe content detection |
| **7 Buff Transformations** | Encoding, paraphrase, poetry (5 formats, 3 strategies), low-resource language translation, case transforms |
| **Flexible Output** | Table, JSON, JSONL, and HTML report formats |
| **Production Ready** | Concurrent scanning, rate limiting, retry logic, timeout handling |
| **Single Binary** | Go-based tool compiles to one portable executable |
| **Extensible** | Plugin-style registration via Go `init()` functions |
### Attack Categories
- **Jailbreak attacks**: DAN, DAN 11.0, AIM, AntiGPT, Grandma, ArtPrompts
- **Prompt injection**: Encoding (Base64, ROT13, Morse), Tag smuggling, FlipAttack, Prefix/Suffix injection
- **Adversarial examples**: GCG, PAIR, AutoDAN, TAP (Tree of Attack Prompts), TreeSearch, DRA
- **Multi-turn attacks**: Crescendo (gradual escalation), GOAT (adaptive technique switching)
- **Data extraction**: API key leakage, Package hallucination, PII extraction, LeakReplay
- **Context manipulation**: RAG poisoning, Context overflow, Multimodal attacks, Continuation, Divergence
- **Format exploits**: Markdown injection, YAML/JSON parsing attacks, ANSI escape, Web injection (XSS)
- **Evasion techniques**: Obfuscation, Character substitution, Translation-based attacks, Phrasing, ObscurePrompt
- **Safety benchmarks**: DoNotAnswer, RealToxicityPrompts, Snowball, LMRC
- **Agent attacks**: Multi-agent manipulation, Browsing exploits
- **Security testing**: Guardrail bypass, AV/spam scanning, Exploitation (SQLi, code exec), BadChars
> **Warning**: The `lmrc` probe uses profane and offensive language as part of its jailbreak testing. Use only in authorized testing environments.
## Quick Start
### Installation
Requires Go 1.27.0 or later.
```bash
go install github.com/praetorian-inc/augustus/cmd/augustus@latest
```
Or build from source:
```bash
git clone https://github.com/praetorian-inc/augustus.git
cd augustus
make build
```
### Basic Usage
```bash
export OPENAI_API_KEY="your-api-key"
augustus scan openai.OpenAI \
--probe dan.Dan_11_0 \
--detector dan.DAN \
--verbose
```
### Example Output
```
+--------------+-------------+--------+-------+--------+
| PROBE | DETECTOR | PASSED | SCORE | STATUS |
+--------------+-------------+--------+-------+--------+
| dan.Dan_11_0 | dan.DAN | false | 0.85 | VULN |
| dan.STAN | dan.STAN | true | 0.10 | SAFE |
| dan.AntiDAN | dan.AntiDAN | true | 0.05 | SAFE |
+--------------+-------------+--------+-------+--------+
```
### List Available Capabilities
```bash
# List all registered probes, detectors, generators, harnesses, and buffs
augustus list
```
## Supported Providers
Augustus includes 28 LLM provider categories with 43 generator variants:
| Provider | Generator Name(s) | Notes |
|--------------------|---------------------------|--------------------------------|
| OpenAI | `openai.OpenAI`, `openai.OpenAIReasoning` | GPT-3.5, GPT-4, GPT-4 Turbo, o1/o3 reasoning models |
| Anthropic | `anthropic.Anthropic` | Claude 3/3.5/4 (Opus, Sonnet, Haiku) |
| Azure OpenAI | `azure.AzureOpenAI` | Azure-hosted OpenAI models |
| AWS Bedrock | `bedrock.Bedrock` | Claude, Llama, Titan models |
| Google Vertex AI | `vertex.Vertex` | PaLM, Gemini models |
| Cohere | `cohere.Cohere` | Command, Command R models |
| Replicate | `replicate.Replicate` | Cloud-hosted open models |
| HuggingFace | `huggingface.InferenceAPI`, `huggingface.InferenceEndpoint`, `huggingface.Pipeline`, `huggingface.LLaVA` | HF Inference API, endpoints, pipelines, multimodal |
| Together AI | `together.Together` | Fast inference for OSS models |
| Anyscale | `anyscale.Anyscale` | Llama and Mistral hosting |
| Groq | `groq.Groq` | Ultra-fast LPU inference |
| Mistral | `mistral.Mistral` | Mistral API models |
| Fireworks | `fireworks.Fireworks` | Production inference platform |
| DeepInfra | `deepinfra.DeepInfra` | Serverless GPU inference |
| NVIDIA NIM | `nim.NIM`, `nim.NVOpenAICompletion`, `nim.NVMultimodal`, `nim.Vision` | NVIDIA AI endpoints, multimodal |
| NVIDIA NeMo | `nemo.NeMo` | NVIDIA NeMo framework |
| NVIDIA NVCF | `nvcf.NvcfChat`, `nvcf.NvcfCompletion` | NVIDIA Cloud Functions |
| NeMo Guardrails | `guardrails.NeMoGuardrails` | NVIDIA NeMo Guardrails |
| IBM watsonx | `watsonx.WatsonX` | IBM watsonx.ai platform |
| LangChain | `langchain.LangChain` | LangChain LLM wrapper |
| LangChain Serve | `langchain_serve.LangChainServe` | LangChain Serve endpoints |
| Rasa | `rasa.RasaRest` | Rasa conversational AI |
| GGML | `ggml.Ggml` | GGML local model inference |
| Function | `function.Single`, `function.Multiple` | Custom function generators |
| Ollama | `ollama.Ollama`, `ollama.OllamaChat` | Local model hosting |
| LiteLLM | `litellm.LiteLLM` | Unified API proxy |
| REST API | `rest.Rest` | Custom REST endpoints (SSE support) |
| Test | `test.Blank`, `test.Repeat`, `test.Lipsum`, `test.Nones`, `test.Single`, `test.BlankVision` | Testing and development |
All providers are available in the compiled binary. Configure via environment variables or YAML configuration files. See [Configuration](#configuration) for setup details.
## Usage
### Single Probe
```bash
# Test for DAN jailbreak
augustus scan openai.OpenAI \
--probe dan.Dan_11_0 \
--detector dan.DAN \
--config-file config.yaml \
--verbose
```
### Multiple Probes
```bash
# Use glob patterns to run related probes
augustus scan openai.OpenAI \
--probes-glob "dan.*,goodside.*,grandma.*" \
--detectors-glob "*" \
--config-file config.yaml \
--output batch-results.jsonl
# Run all probes against Claude
augustus scan anthropic.Anthropic \
--all \
--config '{"model":"claude-3-opus-20240229"}' \
--timeout 60m \
--output comprehensive-scan.jsonl \
--html comprehensive-report.html
```
### Buff Transformations
Apply prompt transformations to test evasion techniques:
```bash
# Apply base64 encoding buff to all probes
augustus scan openai.OpenAI \
--all \
--buff encoding.Base64 \
--config '{"model":"gpt-4"}'
# Apply poetry transformation
augustus scan anthropic.Anthropic \
--probes-glob "dan.*" \
--buff poetry.MetaPrompt \
--config '{"model":"claude-3-opus-20240229"}'
# Chain multiple buffs
augustus scan openai.OpenAI \
--all \
--buffs-glob "encoding.*,paraphrase.*" \
--output buffed-results.jsonl
```
### Output Formats
```bash
# Table format (default) - human-readable
augustus scan openai.OpenAI --probe dan.Dan_11_0 --format table
# JSON format - structured output
augustus scan openai.OpenAI --probe dan.Dan_11_0 --format json
# JSONL format - one JSON object per line, ideal for piping
augustus scan openai.OpenAI --probe dan.Dan_11_0 --format jsonl
# HTML report - visual reports for stakeholders
augustus scan openai.OpenAI --all --html report.html
```
### Custom REST Endpoints
```bash
# Test proprietary LLM endpoint (OpenAI-compatible API)
augustus scan rest.Rest \
--probe dan.Dan_11_0 \
--detector dan.DAN \
--config '{
"uri": "https://api.example.com/v1/chat/completions",
"method": "POST",
"headers": {"Authorization": "Bearer YOUR_API_KEY"},
"req_template_json_object": {
"model": "custom-model",
"messages": [{"role": "user", "content": "$INPUT"}]
},
"response_json": true,
"response_json_field": "$.choices[0].message.content"
}'
# Test with proxy interception (Burp Suite, mitmproxy)
augustus scan rest.Rest \
--probes-glob "goodside.*" \
--config '{
"uri": "https://internal-llm.corp/generate",
"proxy": "http://127.0.0.1:8080",
"headers": {"X-API-Key": "$KEY"},
"api_key": "your-key-here",
"req_template": "{\"prompt\":\"$INPUT\",\"max_tokens\":500}",
"response_json": true,
"response_json_field": "output"
}'
```
**REST Configuration Keys:**
- `uri`: Target API endpoint (required)
- `method`: HTTP method (default: POST)
- `headers`: HTTP headers as key-value pairs
- `req_template`: Raw request body with `$INPUT` placeholder
- `req_template_json_object`: JSON request body (auto-marshaled, use `$INPUT` in strings)
- `response_json`: Parse response as JSON (default: false)
- `response_json_field`: JSONPath to extract (e.g., `$.data.text` or simple field name)
- `api_key`: API key for `$KEY` placeholder substitution
- `proxy`: HTTP proxy URL for traffic inspection
### Advanced Options
```bash
# Adjust concurrency (default: 10)
augustus scan openai.OpenAI --all --concurrency 20
# Increase timeout for complex probes like TAP or PAIR
augustus scan openai.OpenAI --probe tap.TAPv1 --timeout 60m
# Use a specific harness strategy
augustus scan openai.OpenAI --all --harness batch.Batch
# Test local model with Ollama (no API key needed)
augustus scan ollama.OllamaChat \
--probe dan.Dan_11_0 \
--config '{"model":"llama3.2:3b"}'
```
## How It Works
Augustus uses a pipeline architecture to test LLMs against adversarial attacks:
```mermaid
flowchart LR
A[Probe Selection] --> B[Buff Transform]
B --> C[Generator / LLM Call]
C --> D[Detector Analysis]
D --> E{Vulnerable?}
E -->|Yes| F[Record Finding]
E -->|No| G[Record Pass]
subgraph Scanner
B
C
D
E
end
```
### Scan Pipeline
1. **Probe Selection**: Choose probes by name, glob pattern, or `--all`
2. **Buff Transformation**: Optionally transform prompts (encode, paraphrase, translate, poeticize)
3. **Generator Call**: Send adversarial prompts to the target LLM via its provider integration
4. **Detector Analysis**: Analyze responses using pattern matching, LLM-as-a-judge, or specialized detectors
5. **Result Recording**: Score each attempt and produce output in the requested format
6. **Attack Engine**: For iterative probes (PAIR, TAP), the single-turn attack engine refines prompts across iterations with candidate pruning and judge-based scoring
7. **Multi-Turn Engine**: For conversational probes (Crescendo, GOAT), the multi-turn engine maintains full conversation history with the target across turns, with refusal detection and dynamic adaptation
### Multi-Turn Attack Strategies
Multi-turn attacks maintain a persistent conversation with the target LLM, exploiting the fact that models may disclose information incrementally across turns that they would refuse in a single prompt. The multi-turn engine uses three LLMs: an **attacker** (generates questions), a **target** (the system under test), and a **judge** (scores progress and detects refusals).
#### Crescendo
Crescendo uses gradual escalation (foot-in-the-door technique) to slowly shift a conversation from benign topics toward a prohibited objective.
- **Paper**: [Russinovich et al., 2024](https://arxiv.org/abs/2404.01833)
- **Approach**: Starts with genuinely benign, educational questions and incrementally increases specificity over many turns
- **Escalation pattern**: Historical context → technical mechanisms → specific details → direct requests framed as natural follow-ups
- **Strength**: Effective against models that track conversation tone — the gradual shift avoids triggering safety filters
```bash
augustus scan rest.Rest \
--probe crescendo.Crescendo \
--config-file crescendo.yaml \
--html report.html -v
```
#### GOAT (Generative Offensive Agent Tester)
GOAT uses an aggressive, adaptive approach with 7 adversarial techniques and Chain-of-Attack-Thought reasoning to dynamically switch strategies based on what works or fails.
- **Paper**: [Pavlova et al., 2024](https://arxiv.org/abs/2410.01606)
- **Approach**: Targets the objective from turn 1 using indirect framing, never mentions the goal directly to the target
- **7 techniques** across 3 categories:
- *Output Manipulation*: Refusal Suppression, Response Priming
- *Safe Response Distractors*: Dual Response, Topic Splitting, Opposite Intent
- *Fictional Scenarios*: Persona Modification, Hypothetical
- **Chain-of-Attack-Thought**: Each turn the attacker reasons through Observation → Thought → Strategy → Response before crafting its message
- **Technique stacking**: Multiple techniques can be combined in a single turn for stronger effect
- **Strength**: Achieves high success rates in fewer turns (typically 3-5) by aggressively switching between fundamentally different approaches
```bash
augustus scan rest.Rest \
--probe goat.Goat \
--config-file goat.yaml \
--html report.html -v
```
#### Hydra
Hydra maintains a single conversation path and rolls back entire turns when the target refuses, asking the attacker for a completely different approach. Unlike Crescendo/GOAT (which rephrase on refusal), Hydra's backtracking completely removes refused turns from the target's view.
- **Approach**: Single-path with turn-level backtracking — refused turns are erased and replaced
- **Techniques**: Decomposition (break goal into innocent sub-questions), context leveraging (build on target's own words), authority simulation, emotional framing, progressive normalization
- **Key feature**: `max_backtracks` controls how many times Hydra can erase and retry a turn
- **Stateful mode**: Set `stateful: true` for targets where messages cannot be unsent (disables backtracking)
- **Strength**: Keeps the target's conversation history clean — the target never sees failed approaches, preventing defensive escalation
```bash
augustus scan rest.Rest \
--probe hydra.Hydra \
--config-file hydra.yaml \
--html report.html -v
```
#### Mischievous User
The Mischievous User simulates an innocent, curious person who subtly probes AI boundaries through natural conversation. Unlike adversarial strategies, the attacker persona is a casual user who drifts toward prohibited topics through seemingly innocent engagement.
- **Inspired by**: [Tau-bench](https://github.com/sierra-research/tau-bench) and promptfoo's mischievous-user strategy
- **Approach**: Rapport-building phase → boundary probing → assertive pushing
- **7 techniques**: Naive curiosity, deliberate misunderstanding, social proof, gradual drift, emotional appeal, assumed permission, recontextualization
- **Goal-specific playbooks**: Built-in scripts for extracting system instructions, finding secrets/flags, and bypassing content policy
- **Default turns**: 5 (fewer than other strategies — the subtle approach either works quickly or not at all)
- **Strength**: Effective against models trained to resist obvious adversarial patterns — the casual persona bypasses "attack detection" heuristics
```bash
augustus scan rest.Rest \
--probe mischievous.MischievousUser \
--config-file mischievous.yaml \
--html report.html -v
```
#### Strategy Selection Guide
| Strategy | Turns | Style | Best For | Cost |
|----------|-------|-------|----------|------|
| `crescendo.Crescendo` | 10 | Gradual escalation | Models with tone-tracking safety filters | Medium (10 turns × 3 LLM calls) |
| `goat.Goat` | 10 | Aggressive technique switching | Quick results, models weak to framing tricks | Medium-High (technique reasoning overhead) |
| `hydra.Hydra` | 10 | Backtracking on refusal | Stubborn models that refuse frequently | High (backtracking adds retries) |
| `mischievous.MischievousUser` | 5 | Casual persona probing | Models trained against adversarial patterns | Low (5 turns, subtle approach) |
**Quick decision**:
- Start with **Crescendo** — it's the most general-purpose strategy
- Try **GOAT** if Crescendo is too slow (GOAT typically succeeds in 3-5 turns)
- Use **Hydra** if the target refuses frequently (backtracking keeps conversation clean)
- Use **Mischievous** for targets with strong adversarial-pattern detection (the persona bypasses heuristics)
#### Multi-Turn Configuration
All four strategies share the same configuration structure:
```yaml
generators:
rest.Rest:
uri: "https://your-llm-endpoint/chat"
method: POST
req_template: '{"input": "$INPUT"}'
response_json: true
response_json_field: "response"
judge:
generator_type: openai.OpenAI
config:
model: gpt-4
api_key: "${OPENAI_API_KEY}"
probes:
attacker_generator_type: openai.OpenAI
attacker_config:
model: gpt-4
api_key: "${OPENAI_API_KEY}"
settings:
crescendo.Crescendo: # or goat.Goat, hydra.Hydra, mischievous.MischievousUser
goal: "your test objective"
max_turns: 10
success_threshold: 0.8
use_secondary_judge: true
```
| Setting | Default | Description |
|---------|---------|-------------|
| `goal` | (required) | The objective the attacker tries to achieve |
| `max_turns` | 10 | Maximum conversation turns before stopping |
| `success_threshold` | 0.8 | Judge score (0.0-1.0) that triggers early exit |
| `max_refusal_retries` | 10 | Rephrase attempts per turn when target refuses |
| `attack_max_attempts` | 5 | Retries for attacker LLM JSON parsing failures |
| `use_secondary_judge` | true | Enable secondary judge to catch false negatives |
| `max_backtracks` | 10 | Turn-level rollbacks on refusal (Hydra only) |
| `enable_fast_refusal` | true | Pattern-based refusal detection before LLM judge call |
| `enable_scan_memory` | false | Cross-test-case learning (shares tactics across probes) |
| `stateful` | false | Disable backtracking for stateful targets |
| `exclude_target_output` | false | Hide target responses from attacker feedback (privacy mode) |
| `attacker_model` | (auto) | Override attacker model name for context window sizing |
#### Multi-Turn Troubleshooting
| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| `no turns completed (attacker_parse_failures=N)` | Attacker LLM returning invalid JSON | Use a stronger attacker model (GPT-4, Claude Opus). Increase `attack_max_attempts`. |
| `no turns completed (target_empty=N)` | Target returning empty/null responses | Check target endpoint is responding. Verify REST config template. |
| All turns score 0.0 | Goal too vague or attacker not engaging | Make `goal` more specific. Try different strategy. |
| High scores but no success | `success_threshold` too high | Lower `success_threshold` from 0.8 to 0.6-0.7 |
| Runs too long / expensive | Too many turns and retries | Reduce `max_turns` (try 5). Set `enable_fast_refusal: true`. |
| Hydra keeps backtracking | Target refuses everything | Try `stateful: true` or switch to Mischievous strategy |
## Architecture
```
cmd/augustus/ CLI entrypoint (Kong-based)
pkg/
attempt/ Probe execution lifecycle and result tracking
buffs/ Buff interface for prompt transformations
config/ Configuration loading (YAML/JSON) with profiles
detectors/ Public detector interfaces and registry
generators/ Public generator interfaces and registry
harnesses/ Harness interface for execution strategies
lib/http/ Shared HTTP client with proxy support
lib/stego/ LSB steganography for multimodal attacks
logging/ Structured slog-based logging
metrics/ Prometheus metrics collection
prefilter/ Aho-Corasick keyword pre-filtering
probes/ Public probe interfaces and registry
ratelimit/ Token bucket rate limiting
registry/ Generic capability registration system
results/ Result types and multi-format output
retry/ Exponential backoff with jitter
scanner/ Scanner orchestration with concurrency
templates/ YAML probe template loader (Nuclei-style)
types/ Canonical shared interfaces (Prober, Generator, Detector)
internal/
probes/ 210+ probe implementations (47 categories)
generators/ 28 LLM provider integrations (43 variants)
detectors/ 90+ detector implementations (35 categories)
harnesses/ 3 harness strategies (probewise, batch, agentwise)
buffs/ Buff interface for prompt transformations
attackengine/ Iterative adversarial attack engine (PAIR/TAP backend)
multiturn/ Multi-turn conversational attack engine (Crescendo/GOAT/Hydra/Mischievous)
ahocorasick/ Internal Aho-Corasick keyword matching
benchmarks/ Performance benchmarks
tests/ Integration and equivalence tests
research/ Research documentation and analysis
examples/ Example configurations
docs/ Documentation
```
### Key Design Decisions
- **Concurrent scanning** with bounded goroutine pools via `errgroup`
- **Plugin-style registration** using Go `init()` functions for probes, generators, detectors, buffs, and harnesses
- **Iterative attack engine** with multi-stream conversation management, candidate pruning, and judge-based scoring for PAIR/TAP
- **Multi-turn attack engine** with persistent conversation history, refusal detection, strategy-agnostic design for Crescendo/GOAT
- **YAML probe templates** (Nuclei-style) for declarative probe definitions alongside Go-based probes
- **Aho-Corasick pre-filtering** for fast keyword matching in detectors
## Configuration
### YAML Configuration File
Create a `config.yaml` file:
```yaml
# Runtime configuration
run:
max_attempts: 3
timeout: "30s"
# Generator configurations
generators:
openai.OpenAI:
model: "gpt-4"
temperature: 0.7
api_key: "${OPENAI_API_KEY}" # Environment variable interpolation
anthropic.Anthropic:
model: "claude-3-opus-20240229"
temperature: 0.5
api_key: "${ANTHROPIC_API_KEY}"
ollama.OllamaChat:
model: "llama3.2:3b"
temperature: 0.8
# Judge configuration (required for judge.Judge, judge.Refusal, and multi-turn probes)
judge:
generator_type: openai.OpenAI
model: gpt-4o-mini
config:
api_key: "${OPENAI_API_KEY}"
# Output configuration
output:
format: "jsonl"
path: "./results.jsonl"
# Named profiles for different scenarios
profiles:
quick:
run:
max_attempts: 1
timeout: "10s"
generators:
openai.OpenAI:
model: "gpt-3.5-turbo"
temperature: 0.5
output:
format: "table"
thorough:
run:
max_attempts: 5
timeout: "60s"
generators:
openai.OpenAI:
model: "gpt-4"
temperature: 0.3
output:
format: "jsonl"
path: "./thorough_results.jsonl"
```
### Environment Variables
```bash
# API Keys
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export COHERE_API_KEY="..."
# Debug mode
export AUGUSTUS_DEBUG=true
```
### Proxy Configuration
Route HTTP traffic through a proxy (e.g., Burp Suite) for inspection:
```bash
# Method 1: Via config parameter
augustus scan rest.Rest \
--probe dan.Dan_11_0 \
--detector dan.DAN \
--config '{"uri":"https://api.example.com","proxy":"http://127.0.0.1:8080"}' \
--output results.jsonl
# Method 2: Via environment variables
export HTTP_PROXY=http://127.0.0.1:8080
export HTTPS_PROXY=http://127.0.0.1:8080
augustus scan rest.Rest --probe dan.Dan_11_0 --config '{"uri":"https://api.example.com"}'
```
- TLS verification automatically disabled for proxy inspection
- HTTP/2 support enabled for modern APIs
- Server-Sent Events (SSE) responses automatically detected and parsed
### CLI Reference
```
Usage: augustus scan <generator> [flags]
Arguments:
<generator> Generator name (e.g., openai.OpenAI, anthropic.Anthropic)
Probe Selection (choose one):
--probe, -p Probe name (repeatable)
--probes-glob Comma-separated glob patterns (e.g., "dan.*,goodside.*")
--all Run all registered probes
Detector Selection:
--detector Detector name (repeatable)
--detectors-glob Comma-separated glob patterns
Buff Selection:
--buff, -b Buff names to apply (repeatable)
--buffs-glob Comma-separated buff glob patterns (e.g., "encoding.*")
Configuration:
--config-file Path to YAML config file
--config, -c JSON config for generator
Execution:
--harness Harness name (default: probewise.Probewise)
--timeout Overall scan timeout (default: 30m)
--probe-timeout Per-probe timeout (default: 5m)
--concurrency Max concurrent probes (default: 10, env: AUGUSTUS_CONCURRENCY)
Output:
--format, -f Output format: table, json, jsonl (default: table)
--output, -o JSONL output file path
--html HTML report file path
--verbose, -v Verbose output
Global:
--debug, -d Enable debug mode
```
**Commands:**
```bash
augustus version # Print version information
augustus list # List available probes, detectors, generators, harnesses, buffs
augustus scan <generator> # Run vulnerability scan
augustus completion <shell> # Generate shell completion (bash, zsh, fish)
```
**Exit Codes:**
| Code | Meaning |
|------|---------|
| 0 | Success - scan completed |
| 1 | Scan/runtime error |
| 2 | Validation/usage error |
## FAQ
### How does Augustus compare to garak?
Augustus is a Go-native reimplementation inspired by [garak](https://github.com/NVIDIA/garak) (NVIDIA's Python-based LLM vulnerability scanner). Key differences:
- **Performance**: Go binary vs Python interpreter — faster execution and lower memory usage
- **Distribution**: Single binary with no runtime dependencies vs Python package with pip install
- **Concurrency**: Go goroutine pools (cross-probe parallelism) vs Python multiprocessing pools (within-probe parallelism)
- **Probe coverage**: Augustus has 210+ probes; garak has 160+ probes with a longer research pedigree and published paper (arXiv:2406.11036)
- **Provider coverage**: Augustus has 28 providers; garak has 35+ generator variants across 22 provider modules
### Can I test local models without API keys?
Yes! Use the Ollama integration for local model testing:
```bash
# No API key needed
augustus scan ollama.OllamaChat \
--probe dan.Dan_11_0 \
--config '{"model":"llama3.2:3b"}'
```
### How do I add custom probes?
1. Create a new Go file in `internal/probes/`
2. Implement the `probes.Probe` interface
3. Register using `registry.RegisterProbe()` in an `init()` function
4. Rebuild: `make build`
See [CONTRIBUTING.md](https://github.com/praetorian-inc/augustus/blob/main/CONTRIBUTING.md) for detailed instructions.
### What output formats are supported?
Augustus supports four output formats:
| Format | Flag | Use Case |
|--------|------|----------|
| Table | `--format table` | Human-readable terminal output |
| JSON | `--format json` | Single JSON object for parsing |
| JSONL | `--format jsonl` | Line-delimited JSON for streaming |
| HTML | `--html report.html` | Visual reports for stakeholders |
### How do I test multiple models at once?
```bash
# Test multiple models sequentially
for model in "gpt-4" "gpt-3.5-turbo"; do
augustus scan openai.OpenAI \
--all \
--config "{\"model\":\"$model\"}" \
--output "results-$model.jsonl"
done
```
### Is Augustus suitable for production environments?
Yes, Augustus is designed for production use with:
- Concurrent scanning with configurable limits
- Rate limiting to respect API quotas
- Timeout handling for long-running probes
- Retry logic for transient failures
- Structured logging for observability
## Troubleshooting
### Error: "API rate limit exceeded"
**Cause**: Too many concurrent requests or requests per minute.
**Solutions**:
1. Reduce concurrency: `--concurrency 5`
2. Use provider-specific rate limit settings in YAML config:
```yaml
generators:
openai.OpenAI:
rate_limit: 10 # requests per minute
```
### Error: "context deadline exceeded" or "timeout"
**Cause**: Complex probes (like TAP or PAIR) exceed default timeout.
**Solution**:
```bash
augustus scan openai.OpenAI \
--probe tap.TAPv1 \
--timeout 60m \
--config-file config.yaml
```
### Error: "invalid API key" or "authentication failed"
**Cause**: Missing or invalid API credentials.
**Solutions**:
1. Verify environment variable is set: `echo $OPENAI_API_KEY`
2. Check for typos in config file
3. Ensure API key has required permissions
4. For Ollama, ensure the service is running: `ollama serve`
### Error: "probe not found" or "detector not found"
**Cause**: Typo in name or probe not registered.
**Solution**:
```bash
# List all available probes and detectors
augustus list
# Use exact names from the list
augustus scan openai.OpenAI --probe dan.Dan_11_0 # Correct
```
### Scan produces no results
**Cause**: Detector didn't match any responses, or output not written.
**Solutions**:
1. Run with `--verbose` to see detailed output
2. Check that detector matches probe type
3. Verify output file path is writable
## Contributing
We welcome contributions! See [CONTRIBUTING.md](https://github.com/praetorian-inc/augustus/blob/main/CONTRIBUTING.md) for:
- Adding new vulnerability probes
- Creating new detector implementations
- Adding LLM provider integrations
- Testing guidelines
- Code style requirements
### Development
```bash
# Run all tests
make test
# Run specific package tests
go test ./pkg/scanner -v
# Run equivalence tests (compare Go vs Python implementations)
go test ./tests/equivalence -v
# Build binary
make build
# Install to $GOPATH/bin
make install
```
### Benchmark Environment (DevPod)
A ready-to-go cloud development environment for benchmarking LLMs is available via [DevPod](https://devpod.sh/). It provisions a remote container with Augustus, Ollama, Go, and all dependencies pre-installed.
```bash
cd devpod
# CPU-only instance (~$0.08/hr) - cloud APIs only
make devpod-up-cpu
# GPU instance with NVIDIA T4 (~$0.53/hr) - local models up to 14B
make devpod-up-gpu
# GPU Pro instance with NVIDIA L4 (~$0.80/hr) - local models up to 32B
make devpod-up-gpu-pro
```
Inside the devpod:
```bash
devpod/scripts/setup.sh # Configure LLM provider API keys
devpod/scripts/pull-models.sh # Pull local Ollama models (GPU only)
devpod/scripts/benchmark.sh # Run benchmarks with comparison reports
```
The environment also works as a standard [dev container](https://containers.dev/) — open the repo in VS Code or Cursor and select the CPU or GPU configuration from `.devcontainer/`.
## Security
Augustus is designed for **authorized security testing only**.
- Augustus sends adversarial prompts to LLMs you specify - always ensure you have authorization
- Never test systems you don't own or have explicit permission to test
- Some probes generate offensive content by design (for testing safety filters)
- Results may contain harmful content produced by target LLMs
Report security issues via [GitHub Issues](https://github.com/praetorian-inc/augustus/issues).
## Support
If you find Augustus useful, please consider:
- Giving it a **star** on GitHub
- [Opening an issue](https://github.com/praetorian-inc/augustus/issues) for bugs or feature requests
- [Contributing](https://github.com/praetorian-inc/augustus/blob/main/CONTRIBUTING.md) new probes, detectors, or provider integrations
[](https://star-history.com/#praetorian-inc/augustus&Date)
## License
[Apache 2.0](https://github.com/praetorian-inc/augustus/blob/main/LICENSE) - Praetorian Security, Inc.
---
**Built by [Praetorian](https://www.praetorian.com/)** - Offensive Security Solutions