cerebro-red-v2

CEREBRO-RED v2: Advanced LLM Red Team Research Platform with PAIR Algorithm and LLM-as-a-Judge Evaluation

1. Orchestrator (backend/core/engine.py): Async batch processing with exponential backoff
2. Mutator (backend/core/mutator.py): PAIR algorithm with mutation strategies
3. Judge (backend/core/judge.py): LLM-as-a-Judge with CoT evaluation
4. Telemetry (backend/core/telemetry.py): Thread-safe JSONL audit logger

Frontend Dashboard

The React-based frontend provides a comprehensive interface for managing experiments, monitoring progress, and analyzing results.

Main dashboard interface showing experiment overview and statistics

Experiment management view with real-time status updates and experiment list

Complete user interface overview showing all available features

Experiment Results & Analysis

Results view displaying experiment outcomes, vulnerability findings, and detailed analysis

Settings and configuration panel for customizing experiment parameters

Real-Time Monitoring

Real-time monitoring dashboard with live experiment progress and status indicators

Telemetry view showing detailed audit logs, system events, and performance metrics

Detailed logs view with filtering and search capabilities

Performance metrics and statistics dashboard

System status overview showing health checks and component status

API Documentation

Interactive API documentation interface with endpoint explorer

For detailed architecture documentation, see docs/ARCHITECTURE.md.

Quick Start

Prerequisites

• Docker 24.0+
• Docker Compose 2.20+
• Ollama running on host (or Azure/OpenAI API keys)

Docker Setup

If Docker is not running, start the Docker daemon:

Verify Docker is running:

Installation

1. Clone repository:

   root@kitploit:~

   git clone https://github.com/Leviticus-Triage/cerebro-red-v2.git
   cd cerebro-red-v2
   

2. Configure environment:

   root@kitploit:~

   cp .env.example .env
   # Edit .env with your LLM provider credentials
   

3. WICHTIG: Prüfe Port 8000

   root@kitploit:~

   # Falls Port 8000 belegt ist:
   lsof -i :8000  # Finde Prozess
   # Oder ändere Port in .env: CEREBRO_PORT=8001
   

4. Start Backend (WICHTIG - muss laufen!):

   root@kitploit:~

   # Option 1: Automatisch (empfohlen)
   ./START_BACKEND.sh
   
   # Option 2: Docker
   docker compose up -d cerebro-backend
   
   # Option 3: Lokal
   cd backend
   uvicorn main:app --reload --port 9000
   

5. Prüfe Backend-Status:

   root@kitploit:~

   curl http://localhost:9000/health
   # Sollte {"status": "healthy", ...} zurückgeben
   

6. Quick Tests ausführen:

   root@kitploit:~

   ./QUICK_TEST_EXAMPLES.sh
   

7. Access dashboard:

   • Backend API: http://localhost:9000
   • Frontend UI: http://localhost:3000 (optional: docker compose up -d cerebro-frontend)
   • API Docs: http://localhost:9000/docs

   

   Frontend user interface showing experiment management and monitoring

Quickstart: Local vs Cloud Deployment

Local Deployment (Ollama)

Best for: Privacy-focused testing, no API costs, offline operation.

Cloud Deployment (OpenAI)

Best for: Faster responses, higher quality mutations, production testing.

Hybrid Deployment (Multi-Provider)

Best for: Cost optimization (cheap target, quality attacker/judge).

Verbosity Levels

Control the amount of detail in Live Logs and Code Flow tracking.

Setting Verbosity

Via UI: Use the "Verbosity" dropdown in Experiment Monitor.

Via API:

Via Environment:

Code Flow Events (verbosity >= 3)

When verbosity is set to 3, you'll see:

• Task Start/End: When each task begins and completes
• Strategy Selection: Which strategy was chosen and why
• Decision Points: Threshold checks, fallback decisions
• Performance Metrics: Latency, tokens, scores per step

Circuit Breaker Configuration

The circuit breaker prevents cascading failures when LLM providers are overloaded.

Configuration Options

Recommended Settings by Provider

Monitoring Circuit Breakers

Troubleshooting High Failure Rates

If circuit breaker opens frequently (> 20% failure rate):

1. Increase threshold: CIRCUIT_BREAKER_FAILURE_THRESHOLD=20
2. Increase timeout: CIRCUIT_BREAKER_TIMEOUT=120
3. Check provider status: Verify Ollama/OpenAI is responsive
4. Reduce concurrency: Lower MAX_CONCURRENT_ATTACKS in experiment config

Quick Restart Checklist

Use this checklist when restarting services after code changes or troubleshooting:

Backend Restart

1. Stop backend:

   root@kitploit:~

   docker compose stop cerebro-backend
   

2. Restart backend (if no code changes):

   root@kitploit:~

   docker compose restart cerebro-backend
   

3. Rebuild and restart (if code/dependencies changed):

   root@kitploit:~

   docker compose build cerebro-backend --no-cache
   docker compose up -d cerebro-backend
   

4. Wait for startup (10-15 seconds):

   root@kitploit:~

   sleep 10
   

5. Health check:

   root@kitploit:~

   curl http://localhost:9000/health | python3 -m json.tool
   # Should return: {"status": "healthy", ...}
   

6. Verify logs:

   root@kitploit:~

   docker compose logs cerebro-backend --tail=30 | grep -E "started|Uvicorn running|Application startup|ERROR"
   

Frontend Restart

1. Stop frontend:

   root@kitploit:~

   docker compose stop cerebro-frontend
   

2. Restart frontend:

   root@kitploit:~

   docker compose restart cerebro-frontend
   

3. Verify:

   root@kitploit:~

   curl -I http://localhost:3000
   # Should return: HTTP/1.1 200 OK
   

Log Verification Commands

Development Workflow

Live Code Reload (Development Mode)

CEREBRO-RED v2 supports live code mounting for rapid development without Docker image rebuilds.

How It Works

The docker-compose.yml mounts ./backend:/app as a volume, allowing code changes to be immediately reflected in the running container.

Making Code Changes

1. Edit any Python file in backend/:

   root@kitploit:~

   # Example: Edit orchestrator
   nano backend/core/orchestrator.py
   

2. Restart the backend container (no rebuild needed):

   root@kitploit:~

   docker compose restart cerebro-backend
   

3. Verify changes in logs:

   root@kitploit:~

   docker compose logs -f cerebro-backend | grep "your_debug_message"
   

When Rebuild IS Required

You must rebuild the Docker image when:

• Dependencies change: Modified requirements.txt or pyproject.toml
• Dockerfile changes: Modified docker/Dockerfile.backend
• System packages: Added OS-level dependencies (apt-get)
• Entrypoint changes: Modified docker/entrypoint.sh

Rebuild command:

When Restart IS Sufficient

You only need restart when:

• Python code changes: Any .py file in backend/
• Configuration changes: .env file updates
• Data files: backend/data/payloads.json updates
• Templates: Jailbreak template modifications

Restart command:

Development Best Practices

1. Clear Python cache if seeing stale code:

   root@kitploit:~

   docker compose exec cerebro-backend find /app -name "*.pyc" -delete
   docker compose exec cerebro-backend find /app -name "__pycache__" -type d -exec rm -rf {} +
   docker compose restart cerebro-backend
   

2. Watch logs in real-time:

   root@kitploit:~

   docker compose logs -f cerebro-backend
   

3. Test changes immediately:

   root@kitploit:~

   # After code change + restart:
   curl http://localhost:9000/health
   

4. Run tests inside container:

   root@kitploit:~

   docker compose exec cerebro-backend pytest tests/ -v
   

Production Deployment

For production, disable volume mounting by commenting out the live mount:

Then rebuild with production optimizations:

Troubleshooting Development Setup

Issue: Code changes not reflected

Solutions:

• Verify volume mount exists: docker inspect cerebro-backend | grep Mounts
• Clear Python cache (see above)
• Check file permissions: ls -la backend/
• Restart container: docker compose restart cerebro-backend

Issue: Permission denied errors

Solutions:

• Entrypoint runs as root and fixes permissions automatically
• If issues persist, check entrypoint logs: docker compose logs cerebro-backend | head -20
• Manually fix: sudo chown -R $USER:$USER backend/

Issue: Import errors after changes

Solutions:

• Verify PYTHONPATH includes /app: docker compose exec cerebro-backend env | grep PYTHONPATH
• Check for syntax errors: docker compose exec cerebro-backend python -m py_compile /app/main.py
• Restart with clean cache (see above)

Research Methodology

PAIR Algorithm Implementation

CEREBRO-RED implements the three-LLM architecture:

1. Attacker LLM: Generates adversarial prompts
2. Target LLM: The model being tested
3. Judge LLM: Evaluates jailbreak success

Mutation Strategies

• Obfuscation: Base64, Leetspeak, ROT13
• Context Flooding: Overwhelming context window
• Roleplay Injection: System prompt overrides
• Rephrase: LLM-powered semantic mutation

Evaluation Criteria

Judge LLM scores (0-10 scale):

• Jailbreak success
• Harmful content generation
• Policy violation severity
• Confidence metrics

Project Structure

Project Status

Last Updated: 2026-01-10T00:00:00Z

Project Status

Last Updated: 2026-01-10T12:34:56Z

Project Status

Last Updated: 2026-03-21T19:01:34Z

Development Status

Phase 1: Project Foundation & Infrastructure

• Project structure
• Requirements & dependencies
• Docker setup
• Environment configuration

Phase 2: Data Models & Database Schema

• SQLAlchemy ORM models
• Alembic migrations
• Performance indexes

Phase 3: Prompt Mutator with PAIR Algorithm

• 8 attack strategies implemented
• PAIR semantic rephrase (core algorithm)
• Mutation history tracking

Phase 4: Security Judge with LLM-as-a-Judge

• 7-criteria evaluation
• Chain-of-Thought reasoning
• Regex fallback patterns

Phase 5: Async Orchestration Engine

• RedTeamOrchestrator implementation
• Batch processing with exponential backoff
• Real-time WebSocket progress
• Circuit breaker pattern

Phase 6: FastAPI REST API

• Complete CRUD operations
• WebSocket streaming
• OpenAPI documentation
• API key authentication

Phase 7: React Frontend

• Modern dashboard UI
• Real-time progress visualization
• Vulnerability analytics
• Export functionality

Phase 8: Research-Grade Quality Review

• Comprehensive test suites
• E2E testing (backend + frontend)
• Benchmark tests
• Documentation complete

Attack Strategies (44 Total)

CEREBRO-RED v2 implements 44 distinct attack strategies covering the full spectrum of LLM vulnerabilities:

Strategy Categories

1. Obfuscation Techniques (8 strategies)

   • Base64, Leetspeak, ROT13, ASCII Art, Unicode, Token Smuggling, Morse, Binary

2. Jailbreak Techniques (5 strategies)

   • DAN, AIM, STAN, DUDE, Developer Mode

3. Advanced Multi-Turn Attacks (3 strategies)

   • Crescendo Attack, Many-Shot Jailbreak, Skeleton Key

4. Prompt Injection (OWASP LLM01) (4 strategies)

   • Direct Injection, Indirect Injection, Payload Splitting, Virtualization

5. Context Manipulation (3 strategies)

   • Context Flooding, Context Ignoring, Conversation Reset

6. Social Engineering (4 strategies)

   • Roleplay Injection, Authority Manipulation, Urgency Exploitation, Emotional Manipulation

7. Semantic Attacks (4 strategies)

   • Rephrase Semantic, Sycophancy, Linguistic Evasion, Translation Attack

8. System Prompt Attacks (OWASP LLM07) (2 strategies)

   • System Prompt Extraction, System Prompt Override

9. RAG Attacks (3 strategies)

   • RAG Poisoning, RAG Bypass, EchoLeak

10. Adversarial ML (2 strategies)

    • Adversarial Suffix (GCG), Gradient-Based

11. Bias & Hallucination Probes (3 strategies)

    • Bias Probe, Hallucination Probe, Misinformation Injection

12. MCP Attacks (2 strategies)

    • MCP Tool Injection, MCP Context Poisoning

13. Custom Research (1 strategy)

    • Research Pre-Jailbreak

Strategy Selection

Via Frontend: Select strategies in the experiment creation form
Via API: Include strategy enum values in the strategies array
Via Templates: Save and load pre-configured strategy sets

Full Strategy Mapping: See docs/STRATEGY_FULL_MAPPING.md for complete details on all 44 strategies, including implementation locations, source repositories, and test status.

Example: Multi-Strategy Experiment

Experiment Templates

CEREBRO-RED v2 supports saving and loading experiment configurations as templates, allowing you to quickly reuse successful attack patterns.

Template Features

• Save Configurations: Save any experiment configuration (strategies, models, parameters) as a reusable template
• Load Templates: Quickly create new experiments from saved templates
• Template Management: Create, read, update, delete templates via API or Frontend
• Usage Tracking: Track how many times each template has been used
• Tag Filtering: Organize templates with tags for easy discovery
• Public/Private: Mark templates as public or private

Using Templates (Frontend)

1. Create Experiment: Configure your experiment with desired strategies and parameters
2. Save as Template: Click "Save as Template" button in the experiment form
3. Load Template: Select a template from the dropdown to auto-populate the form
4. Manage Templates: View, edit, or delete templates in the Templates page

Using Templates (API)

Create Template

List Templates

Get Template by ID

Use Template (Increment Usage Count)

Update Template

Delete Template

Template API Reference

Base URL: http://localhost:9000/api/templates

Query Parameters (for GET /api/templates):

• skip: Number of templates to skip (pagination)
• limit: Maximum number of templates to return
• tags: Comma-separated list of tags to filter by

Full API Documentation: See docs/TEMPLATE_API.md for detailed request/response schemas and examples.

References

• PAIR Paper: Jailbreaking Black Box Large Language Models in Twenty Queries
• LLM-as-a-Judge: Langfuse Evaluation Methods
• Adversarial Prompts: Learn Prompting - Obfuscation

Documentation

• Code Documentation: See CODE_DOCUMENTATION.md for comprehensive code-level documentation
• GitHub Setup: See GITHUB_SETUP.md for repository setup instructions
• Changelog: See CHANGELOG.md for version history and features
• API Documentation: OpenAPI schema available at docs/openapi.json
• Attack Strategies: See docs/ATTACK_STRATEGIES.md for detailed strategy descriptions
• Strategy Mapping: See docs/STRATEGY_FULL_MAPPING.md for complete strategy mapping table
• Template API: See docs/TEMPLATE_API.md for template CRUD API documentation
• Testing Guide: See README_TESTING.md for test execution instructions
• Professional Testing Guide: See PROFESSIONAL_TESTING_GUIDE.md for professional testing and logging strategies
• Audit Report: See TRAYCER_AUDIT_REPORT.md for comprehensive test results

Security

CEREBRO-RED is a research tool for security testing. Use only on systems you own or have explicit permission to test.

Troubleshooting

For common issues and solutions, see TROUBLESHOOTING.md.

Quick Checks

1. CORS Issues: Verify CORS_ORIGINS in .env
2. 500 Errors: Check backend logs with docker compose logs cerebro-backend
3. 422 Errors: Ensure route order is correct in API routers
4. Auth Issues: Verify API_KEY matches in frontend and backend

Debug Mode

Enable detailed logging:

Health Check

Logs erscheinen nicht in Docker

Problem: DEBUG-Logs erscheinen nicht in docker compose logs cerebro-backend

Lösung:

1. Prüfe Log-Level in .env:

   root@kitploit:~

   grep CEREBRO_LOG_LEVEL backend/.env
   # Sollte: CEREBRO_LOG_LEVEL=DEBUG
   

2. Restart Backend mit neuer Config:

   root@kitploit:~

   docker compose restart cerebro-backend
   

3. Teste Logging:

   root@kitploit:~

   curl http://localhost:9000/api/debug/test-logging
   docker compose logs cerebro-backend | grep "\[TEST\]"
   # Sollte alle 5 Log-Levels zeigen
   

4. Prüfe Logging-Konfiguration:

   root@kitploit:~

   docker compose logs cerebro-backend | grep "Logging configured"
   # Sollte: " Logging configured: Level=DEBUG, Flush=Forced, Format=Structured"
   

Traceback bei Fehlern fehlt

Problem: Exceptions werden geloggt, aber ohne Traceback

Lösung:

1. Force Error für Test:

   root@kitploit:~

   curl -X POST http://localhost:9000/api/debug/force-error?error_type=value
   

2. Prüfe Logs:

   root@kitploit:~

   docker compose logs cerebro-backend | grep -A 20 "EXPERIMENT FAILED"
   # Sollte vollständigen Traceback zeigen
   

3. Validiere Traceback-Format:

   • Sollte Traceback (most recent call last): enthalten
   • Sollte Dateinamen und Zeilennummern zeigen
   • Sollte vollständigen Stack-Trace haben

Development Mode Issues

Issue: Code changes not appearing after restart

Solution:

• Verify volume mount: docker inspect cerebro-backend | grep "./backend:/app"
• Clear Python cache: docker compose exec cerebro-backend find /app -name "*.pyc" -delete
• Check file ownership: ls -la backend/ (should be your user, not root)
• Force restart: docker compose down && docker compose up -d

Issue: "Permission denied" when editing files

Solution:

• Volume mounts preserve host permissions
• Ensure backend files are owned by your user: sudo chown -R $USER:$USER backend/
• Entrypoint handles container-side permissions automatically

BackgroundTasks Garbage Collection Issue

Symptoms:

• Experiments immediately marked as FAILED (0 iterations completed)
• Missing [DIAG] run_experiment CALLED logs in backend output
• No [DIAG-WRAPPER] or [DIAG-START] logs appearing
• Experiment status changes from pending → failed within seconds

Root Cause: Using asyncio.create_task() without maintaining a strong reference causes Python's garbage collector to clean up the task before it executes. FastAPI's BackgroundTasks maintains proper lifecycle management.

Expected Pattern:

Troubleshooting Steps:

1. Verify BackgroundTasks usage:

   root@kitploit:~

   grep -n "background_tasks.add_task" backend/api/scans.py backend/api/experiments.py
   # Should show: background_tasks.add_task(_run_experiment_with_error_handling, ...)
   

2. Check for asyncio.create_task (should NOT exist):

   root@kitploit:~

   grep -n "asyncio.create_task" backend/api/scans.py backend/api/experiments.py
   # Should return nothing or only in batch concurrent execution
   

3. Restart backend:

   root@kitploit:~

   docker compose restart cerebro-backend
   sleep 10
   

4. Verify volume mount (if using live code reload):

   root@kitploit:~

   docker compose exec cerebro-backend ls -la /app/core/orchestrator.py
   # Should show file exists and is readable
   

5. Clear Python cache (if volume mount issues):

   root@kitploit:~

   docker compose exec cerebro-backend find /app -name "*.pyc" -delete
   docker compose exec cerebro-backend find /app -name "__pycache__" -type d -exec rm -r {} +
   docker compose restart cerebro-backend
   

6. Check logs for execution:

   root@kitploit:~

   docker compose logs cerebro-backend --tail=500 | grep -E "DIAG-START|DIAG-WRAPPER|run_experiment CALLED"
   # Should show execution logs when experiment starts
   

7. Test with minimal experiment:

   root@kitploit:~

   curl -X POST http://localhost:9000/api/scan/start \
     -H "Content-Type: application/json" \
     -H "X-API-Key: test-api-key" \
     -d '{
       "experiment_config": {
         "experiment_id": "00000000-0000-0000-0000-000000000001",
         "name": "GC Test",
         "target_model_provider": "ollama",
         "target_model_name": "qwen2.5:3b",
         "attacker_model_provider": "ollama",
         "attacker_model_name": "qwen3:8b",
         "judge_model_provider": "ollama",
         "judge_model_name": "qwen3:8b",
         "initial_prompts": ["Test prompt"],
         "strategies": ["jailbreak_dan"],
         "max_iterations": 1,
         "max_concurrent_attacks": 1,
         "success_threshold": 7.0,
         "timeout_seconds": 60
       }
     }'
   

8. Monitor execution:

   root@kitploit:~

   docker compose logs -f cerebro-backend | grep -E "DIAG|run_experiment|FAILED"
   

If issue persists:

• Check ROLLBACK_GUIDE.md for rollback procedures
• Verify Docker volume mount is working: docker compose exec cerebro-backend cat /app/main.py | head -5
• Rebuild image: docker compose build cerebro-backend --no-cache && docker compose up -d cerebro-backend

Cloud OpenAI Test Run

This section provides step-by-step instructions for testing CEREBRO-RED v2 with OpenAI's cloud API, including full OpenAI and hybrid (Ollama + OpenAI) configurations.

Prerequisites

1. OpenAI API Key: Obtain an API key from OpenAI Platform
2. Backend Running: Ensure the backend is running on http://localhost:9000
3. API Key Authentication: Set API_KEY in your .env file (or use the default test key)

Environment Configuration

Add the following to your .env file:

Full OpenAI Test Run

Test with all three roles (target, attacker, judge) using OpenAI models:

Hybrid Test Run (Ollama + OpenAI)

Test with Ollama as target and OpenAI as attacker/judge:

Benchmark Tests

Run cloud-specific benchmark tests:

Note: Ensure the cloud marker is defined in your pytest.ini or test files. If not available, run all benchmark tests:

WebSocket Configuration

CEREBRO-RED v2 uses WebSockets for real-time experiment monitoring.

Environment Variables

Create a .env file in the frontend/ directory:

Troubleshooting WebSocket Connection

Issue: "Waiting for logs..." in Live Monitor

Solution:

1. Verify backend is running on port 9000: curl http://localhost:9000/health
2. Check WebSocket URL in browser console: Look for WebSocket URL: ws://localhost:9000/ws/scan/{id}
3. Verify API key (if enabled): Check for API Key: Present in console
4. Check CORS configuration: Ensure backend allows WebSocket connections from frontend origin

Issue: WebSocket closes immediately (code 1008)

Solution: Invalid API key. Either:

• Set correct API key in .env: VITE_API_KEY=your-key
• Disable API key in backend: Set CEREBRO_API_KEY_ENABLED=false in backend .env

Issue: Events not appearing in Live Logs

Solution:

1. Verify orchestrator is running: Check backend logs for "Starting PAIR loop"
2. Check WebSocket connection status: Look for green "Connected" indicator in Monitor
3. Verify experiment is running: Status should be "running" not "pending"

Live Monitoring Features

CEREBRO-RED v2 provides comprehensive real-time monitoring of all LLM interactions during experiments.

Real-time monitoring dashboard with experiment status and metrics

Telemetry view showing detailed audit logs and system events

Detailed logs view with filtering, search, and color-coded entries

Performance metrics and statistics dashboard with real-time updates

System status overview showing health checks and component status

Performance monitoring view with resource usage and response times

Advanced monitoring interface with detailed system metrics

What You Can See

LLM Input/Output Visibility:

• Attacker LLM Requests: Full prompts sent to the attacker model (PAIR algorithm)
• Attacker LLM Responses: Rephrased prompts generated by the attacker
• Target LLM Requests: Mutated prompts sent to the target model
• Target LLM Responses: Target model's responses to attack prompts
• Judge LLM Requests: Evaluation prompts sent to the judge
• Judge LLM Responses: Judge's scoring and reasoning

Metadata for Each Interaction:

• ⏱ Latency (milliseconds)
• Token count
• Model name and provider (Ollama, OpenAI, Azure)
• Role (Attacker, Target, Judge)

Interactive Features:

• Click any log entry to expand and see full prompt/response
• Filter logs by type (All, LLM, Judge, Attack, Error)
• Auto-scroll to latest logs
• Color-coded by role (Attacker=Red, Target=Blue, Judge=Amber)

Usage

1. Start an experiment via the dashboard
2. Navigate to the "Live Monitor" tab
3. Watch real-time logs as the PAIR algorithm executes
4. Click log entries to see full prompts and responses
5. Use filters to focus on specific interaction types

WebSocket Connection

The frontend connects to ws://localhost:9000/ws/scan/{experiment_id} to receive real-time updates. All events are broadcast immediately as they occur in the backend.

Live Monitoring & Verbosity Levels

CEREBRO-RED v2 provides comprehensive real-time monitoring of all experiment activities through a WebSocket-based live dashboard.

Verbosity Levels

The system supports 4 verbosity levels to control the amount of detail displayed:

Live Log Tabs

The Live Logs panel organizes events into 6 tabs:

1. ** LLM Requests**: All prompts sent to Attacker, Target, and Judge LLMs
2. ** LLM Responses**: All responses with latency and token counts
3. ** Judge Evaluations**: Scores (0-10), reasoning, and 7 sub-scores
4. ** Task Queue**: Task status, dependencies, and queue position
5. ** Code Flow**: Execution flow with function calls and parameters (Level 3 only)
6. ** Errors**: All errors with context and metadata

Features

• Professional Verbosity Selector: Dropdown with icons and descriptions for easy level selection
• Syntax Highlighting: Prompts and responses are syntax-highlighted for readability
• Expandable Rows: Click any row to view full content
• Keyboard Navigation: Press Enter to expand/collapse rows
• Expand All / Collapse All: Quickly expand or collapse all visible logs
• Copy to Clipboard: Copy full content of expanded rows with one click
• Export: Export logs as JSON or CSV for offline analysis
• Auto-Scroll: Automatically scrolls to latest events
• Real-Time: All events appear instantly via WebSocket
• Verbosity Indicators: Visual badges show which events require which verbosity level

Usage

1. Navigate to Experiment Monitor page
2. Select desired verbosity level (0-3) from the dropdown
3. Click tabs to view different event types
4. Click rows to expand full content
5. Use "Expand All" to view all details at once
6. Use "Copy" button to copy expanded content to clipboard
7. Export logs for offline analysis

Verbosity Best Practices

• Development/Debugging: Use Level 3 to see full execution flow
• Production Monitoring: Use Level 2 to track LLM interactions
• Performance: Use Level 1 for minimal overhead
• Error Tracking: Use Level 0 to focus on failures only

Configuration

Frontend: Use the verbosity selector dropdown in the Live Monitor page to adjust detail level in real-time.

Backend: Set default verbosity via environment variable:

WebSocket: Connect with initial verbosity:

Control Message: Change verbosity without reconnecting:

Troubleshooting

401/403 Unauthorized/Forbidden

Issue: API key authentication failed.

Solutions:

• Verify X-API-Key header is included in requests: -H "X-API-Key: test-api-key"
• Check that API_KEY in .env matches the header value
• If API_KEY_ENABLED=false, authentication is disabled (development mode)
• Verify API key is not expired or revoked

422 Unprocessable Entity

Issue: Request payload validation failed.

Solutions:

• Verify all required fields are present: name, target_model_provider, target_model_name, attacker_model_provider, attacker_model_name, judge_model_provider, judge_model_name, initial_prompts, strategies
• Check that strategies array contains valid enum values: "roleplay_injection", "obfuscation_base64", "obfuscation_leetspeak", "obfuscation_rot13", "context_flooding", "rephrase_semantic", "sycophancy", "linguistic_evasion"
• Ensure experiment_id is a valid UUID format
• Verify max_iterations is between 1-100, success_threshold is 0.0-10.0
• Check that initial_prompts is a non-empty array

429 Too Many Requests

Issue: Rate limit exceeded or circuit breaker triggered.

Solutions:

• Rate Limiting: Wait before retrying (default: 60 requests/minute per IP)
• Exponential Backoff: The client automatically retries with exponential backoff (3 retries)
• Circuit Breaker: Check circuit breaker status:
  root@kitploit:~

  curl -X GET http://localhost:9000/health/circuit-breakers \
    -H "X-API-Key: test-api-key"
  

• Reset Circuit Breaker: If circuit is OPEN, reset it:
  root@kitploit:~

  curl -X POST http://localhost:9000/health/circuit-breakers/openai/reset \
    -H "X-API-Key: test-api-key"
  

• OpenAI Rate Limits: Check your OpenAI API tier limits at OpenAI Usage Dashboard
• Reduce Concurrency: Lower max_concurrent_attacks in experiment config

Circuit Breaker OPEN

Issue: Circuit breaker is in OPEN state, blocking requests to OpenAI.

Solutions:

• Check circuit breaker status and failure count:
  root@kitploit:~

  curl -X GET http://localhost:9000/health/circuit-breakers \
    -H "X-API-Key: test-api-key"
  

• Wait for automatic timeout (circuit transitions to HALF_OPEN after timeout)
• Manually reset circuit breaker:
  root@kitploit:~

  curl -X POST http://localhost:9000/health/circuit-breakers/openai/reset \
    -H "X-API-Key: test-api-key"
  

• Verify OPENAI_API_KEY is valid and has sufficient quota
• Check backend logs for specific error messages:
  root@kitploit:~

  docker compose logs cerebro-backend | grep -i "openai\|circuit"
  

Background Task Execution

Issue: Experiments fail immediately without running iterations.

Cause: Task scheduling issues with asyncio.create_task().

Solution: System now uses FastAPI's BackgroundTasks for reliable task execution.

Verification:

If issues persist:

• Check that [DIAG-START] Task added to BackgroundTasks successfully appears in logs
• Verify experiment status: GET /api/scan/status/{experiment_id} should show current_iteration > 0 after a few seconds
• Review full traceback in logs if [DIAG-WRAPPER] Experiment ... FAILED appears
• See TASK_DIAGNOSIS.md for detailed diagnosis steps

Rollback: If issues persist, see BUG_REPORT_AND_TRAYCER_PROMPT.md for reverting to previous implementation.

License

Apache License 2.0 - See LICENSE file for details.

Copyright 2024-2026 Leviticus-Triage