grubcrawler

Agentic web crawler with anti-detection, vision fallback, and peer-to-peer mesh.

Endpoints · Mesh · Anti-Detection · Ghost Protocol · Live Stream · MCP Tools · Quick Start · Benchmarks · Architecture

---

Full-stack web crawling engine with JavaScript rendering, Camoufox anti-detect browser, per-request proxy routing, and autonomous agent loops. Converts pages to clean markdown with a native Rust extraction engine. When standard crawling is blocked by Cloudflare, CAPTCHAs, or JavaScript walls, Ghost Protocol captures a screenshot and extracts content via vision AI (Claude, GPT-4o, or Ollama). Supports multi-provider LLM orchestration across OpenAI, Anthropic, and Ollama in a single session. Nodes coordinate over a gossip-based peer-to-peer mesh for distributed crawling.

---

Why Grub

We integrated features from every major crawler — then added what none of them have.

Self-Hosted Crawlers

Cloud / Managed Crawlers

Only Grub has Ghost Protocol — automatic vision-based fallback that screenshots blocked pages and extracts content via LLM when standard crawling fails. Prevention (Camoufox + proxy + stealth) handles 95% of blocks. Ghost Protocol handles the rest.

API Endpoints

Core Crawling

Agent (Mode B)

Job Management

Remote Cache

Session Management

Live Stream

Mesh

System

MCP Tools (grub-crawl.py)

The MCP bridge exposes all capabilities to any MCP-compatible host:

Internal Modules

Agent Core (app/agent/)

Provider Adapters (app/agent/providers/)

Policy Gates (app/policy/)

Observability (app/observability/)

API Layer

Anti-Detection (app/)

Mesh (app/mesh/)

Infrastructure

Agent State Machine

INIT -> PLAN -> EXECUTE_TOOL -> OBSERVE -> PLAN -> ... -> RESPOND -> STOP
                     |                                        |
                     +-- policy_denied ---------------------->+
                     +-- max_steps / max_wall_time / max_failures -> STOP
                     +-- no_op_loop (3x empty) ------------> STOP
                     +-- blocked (ghost trigger) -----------> GHOST -> OBSERVE

Stop conditions enforced every iteration:

• max_steps (default: 12)
• max_wall_time (default: 90s)
• max_failures (default: 3)
• no_op_loop (3 consecutive empty responses)
• policy_denied (blocked tool/domain)
• completed (agent responds with text)

Anti-Detection

Three layers of anti-detection that stack together. Prevention stops blocks before they happen. Ghost Protocol handles them after.

Camoufox Engine

Pluggable anti-detect browser with C++-level fingerprint spoofing. No manual user-agent tricks — Camoufox generates realistic fingerprints per context at the browser level, including canvas, WebGL, fonts, and navigator properties.

# Switch engine (default: chromium)
BROWSER_ENGINE=camoufox

Per-Request Proxy

Route crawl traffic through residential, datacenter, or custom proxy pools. Per-request override with env-based defaults. Full Playwright-compatible proxy config.

# Env-based default
PROXY_SERVER=http://proxy.example.com:10001
PROXY_USERNAME=your_username
PROXY_PASSWORD=your_password

# Or per-request
curl -X POST http://localhost:6792/api/crawl \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "options": {
      "proxy": {
        "server": "http://proxy.example.com:10001",
        "username": "your_username",
        "password": "your_password"
      }
    }
  }'

Stealth Mode

Opt-in playwright-stealth patches for Chromium (skipped for Camoufox where it's built-in). Blocks 20+ tracking/analytics domains (Google Analytics, DataDome, PerimeterX, etc.) to reduce fingerprint surface.

STEALTH_ENABLED=true
BLOCK_TRACKING_DOMAINS=true

Ghost Protocol

When a crawl result signals an anti-bot block (Cloudflare challenge, CAPTCHA, empty SPA shell), the agent can switch to cloak mode:

1. Take a full-page screenshot via Playwright
2. Send the image to a vision-capable LLM (Claude Sonnet or GPT-4o)
3. Extract content from the rendered pixels
4. Return extracted text with render_mode: "ghost" in the trace

This bypasses DOM-based anti-bot detection entirely.

Requires AGENT_GHOST_ENABLED=true. Auto-triggers on detected blocks when AGENT_GHOST_AUTO_TRIGGER=true.

Mesh

Agents talking to agents. Every Grub instance is both a worker and a coordinator. Local node offloads to cloud, cloud delegates to local. Tool calls cross the wire transparently.

Node A (local)                    Node B (cloud)
┌─────────────┐                  ┌─────────────┐
│ AgentEngine  │                  │ AgentEngine  │
│     ↓        │                  │     ↓        │
│ MeshDispatcher ──── HTTP ────→ MeshDispatcher │
│     ↓        │                  │     ↓        │
│ Dispatcher   │                  │ Dispatcher   │
│     ↓        │                  │     ↓        │
│ ToolRegistry │                  │ ToolRegistry │
└─────────────┘                  └─────────────┘
       ↕ heartbeat (15s)                ↕
       └────────────────────────────────┘

How it works:

• Discovery — nodes join via seed peer list, then gossip (1-hop) to learn about others
• Heartbeat — every 15s, nodes exchange load metrics. 3 missed = unhealthy. 2 min = removed
• Routing — MeshDispatcher scores all nodes by load, locality, and affinity, then routes tool calls to the best node
• 1-hop max — Node A → B only, never A → B → C. Prevents routing loops
• Local fallback — if remote execution fails, falls back to local Dispatcher
• HMAC auth — all mesh traffic is signed with a shared secret (SHA-256, 60s TTL)

Run a 2-Node Mesh Locally

# Docker Compose (recommended)
./deploy.sh mesh           # Linux/Mac
./deploy.ps1 -Target mesh  # Windows

# Verify
curl http://localhost:6792/mesh/peers  # Node A sees Node B
curl http://localhost:6793/mesh/peers  # Node B sees Node A

Connect Local to Cloud Run

# Deploy to Cloud Run with mesh
./deploy.sh cloudrun latest --mesh-peer http://your-local-ip:6792 --mesh-secret mysecret

# Start local node
MESH_ENABLED=true MESH_SECRET=mysecret MESH_PEERS=https://your-cloud-run-url \
  MESH_ADVERTISE_URL=http://your-local-ip:6792 \
  uvicorn app.main:app --port 6792

Manual Setup

# Node A
MESH_ENABLED=true MESH_NODE_NAME=local MESH_SECRET=test123 \
  MESH_ADVERTISE_URL=http://localhost:6792 \
  uvicorn app.main:app --port 6792

# Node B
MESH_ENABLED=true MESH_NODE_NAME=cloud MESH_SECRET=test123 \
  MESH_PEERS=http://localhost:6792 \
  MESH_ADVERTISE_URL=http://localhost:8081 \
  uvicorn app.main:app --port 8081

When mesh is disabled (MESH_ENABLED=false, the default), Grub operates as a normal single-node crawler with zero mesh overhead.

Live Stream

Watch the crawler work in real-time. A persistent pool of warm Chromium instances streams viewport frames over WebSocket or MJPEG.

WebSocket — connect and send interactive commands:

const ws = new WebSocket("ws://localhost:6792/stream/my-session?url=https://example.com");
ws.onmessage = (e) => {
  const msg = JSON.parse(e.data);
  if (msg.type === "frame") document.getElementById("viewport").src = "data:image/jpeg;base64," + msg.data;
};
// Navigate, click, scroll, type — all over the same socket
ws.send(JSON.stringify({ action: "navigate", url: "https://example.com/pricing" }));
ws.send(JSON.stringify({ action: "click", selector: "#signup-btn" }));
ws.send(JSON.stringify({ action: "scroll", direction: "down" }));

MJPEG — drop it in an `` tag, instant video:

<img src="http://localhost:6792/stream/my-session/mjpeg?url=https://example.com" />

Requires BROWSER_STREAM_ENABLED=true. Each Chromium instance uses ~150-300MB RAM.

Quick Start

Local Development

git clone <repo>
cd grub-crawl
cp .env.example .env
pip install -r requirements.txt
uvicorn app.main:app --reload --host 0.0.0.0 --port 6792

Enable Agent Mode B

# Add to .env
AGENT_ENABLED=true
OPENAI_API_KEY=sk-...
# or
ANTHROPIC_API_KEY=sk-ant-...
AGENT_PROVIDER=anthropic

Submit an Agent Task

curl -X POST http://localhost:6792/api/agent/run \
  -H "Content-Type: application/json" \
  -d '{
    "task": "Find the pricing page on example.com and extract plan details",
    "max_steps": 10,
    "allowed_domains": ["example.com"]
  }'

Docker

# Single node
./deploy.sh local            # or ./deploy.ps1 -Target local

# 2-node mesh
./deploy.sh mesh             # or ./deploy.ps1 -Target mesh

# Cloud Run
./deploy.sh cloudrun v1.0.0  # or ./deploy.ps1 -Target cloudrun -Tag v1.0.0

# Cloud Run + mesh (connect to local node)
./deploy.sh cloudrun v1.0.0 --mesh-peer http://your-ip:6792 --mesh-secret mykey

Anti-Detection (Camoufox + Proxy)

# Add to .env
BROWSER_ENGINE=camoufox
STEALTH_ENABLED=true
BLOCK_TRACKING_DOMAINS=true

# Optional: proxy
PROXY_SERVER=http://proxy.example.com:10001
PROXY_USERNAME=your_username
PROXY_PASSWORD=your_password

Ghost Protocol (anti-bot bypass)

# Add to .env
AGENT_GHOST_ENABLED=true

curl -X POST http://localhost:6792/api/agent/ghost \
  -H "Content-Type: application/json" \
  -d '{"url": "https://blocked-site.com"}'

Live Browser Stream

# Add to .env
BROWSER_STREAM_ENABLED=true
BROWSER_POOL_SIZE=2

# MJPEG (open in browser)
open "http://localhost:6792/stream/demo/mjpeg?url=https://example.com"

Configuration

Server

• HOST (default: 0.0.0.0)
• PORT (default: 6792)
• DEBUG (default: false)

Storage

• STORAGE_PATH (default: ./storage)
• RUNNING_IN_CLOUD (default: false)
• GCS_BUCKET_NAME
• GOOGLE_CLOUD_PROJECT

Authentication

• DISABLE_AUTH (default: false)
• GNOSIS_AUTH_URL (default: http://gnosis-auth:5000)

Browser Engine

• BROWSER_ENGINE — chromium | camoufox (default: chromium)

Crawling

• MAX_CONCURRENT_CRAWLS (default: 5)
• CRAWL_TIMEOUT (default: 30)
• ENABLE_JAVASCRIPT (default: true)
• ENABLE_SCREENSHOTS (default: false)

Proxy

• PROXY_SERVER — proxy URL (e.g. http://proxy:10001)
• PROXY_USERNAME
• PROXY_PASSWORD
• PROXY_BYPASS — comma-separated bypass list

Stealth

• STEALTH_ENABLED (default: false) — playwright-stealth patches
• BLOCK_TRACKING_DOMAINS (default: false) — block analytics/tracking requests

Agent (Mode B)

• AGENT_ENABLED (default: false)
• AGENT_MAX_STEPS (default: 12)
• AGENT_MAX_WALL_TIME_MS (default: 90000)
• AGENT_MAX_FAILURES (default: 3)
• AGENT_ALLOWED_TOOLS — comma-separated allowlist
• AGENT_ALLOWED_DOMAINS — comma-separated allowlist
• AGENT_BLOCK_PRIVATE_RANGES (default: true)
• AGENT_REDACT_SECRETS (default: true)

LLM Providers

• AGENT_PROVIDER — openai | anthropic | ollama (default: openai)
• OPENAI_API_KEY
• OPENAI_MODEL (default: gpt-4.1-mini)
• ANTHROPIC_API_KEY
• ANTHROPIC_MODEL (default: claude-3-5-sonnet-latest)
• OLLAMA_BASE_URL (default: http://localhost:11434)
• OLLAMA_MODEL (default: llama3.1:8b-instruct)

Ghost Protocol

• AGENT_GHOST_ENABLED (default: false)
• AGENT_GHOST_AUTO_TRIGGER (default: true)
• AGENT_GHOST_VISION_PROVIDER — inherits from AGENT_PROVIDER
• AGENT_GHOST_MAX_IMAGE_WIDTH (default: 1280)

Mesh

• MESH_ENABLED (default: false) — master switch
• MESH_PEERS — comma-separated seed peer URLs
• MESH_NODE_NAME — human-readable name (default: hostname)
• MESH_SECRET — shared HMAC secret for inter-node auth
• MESH_ADVERTISE_URL — URL peers use to reach this node
• MESH_PREFER_LOCAL (default: true) — bias toward local execution
• MESH_HEARTBEAT_INTERVAL_S (default: 15)
• MESH_PEER_TIMEOUT_S (default: 45) — mark unhealthy after this
• MESH_PEER_REMOVE_S (default: 120) — remove from peer table after this
• MESH_REMOTE_TIMEOUT_MS (default: 35000) — timeout for remote tool calls

Live Stream

• BROWSER_POOL_SIZE (default: 1)
• BROWSER_STREAM_ENABLED (default: false)
• BROWSER_STREAM_QUALITY (default: 25) — JPEG quality 1-100
• BROWSER_STREAM_MAX_WIDTH (default: 854)
• BROWSER_STREAM_MAX_LEASE_SECONDS (default: 300)

Response Contract

POST /api/markdown returns:

success, url, final_url, status_code, markdown, markdown_plain, content, render_mode, wait_strategy, timings_ms, blocked, block_reason, captcha_detected, http_error_family, body_char_count, body_word_count, , , , , , , , , ,

Content Quality

• blocked — anti-bot/captcha/challenge
• empty — very low signal
• minimal — thin/error pages
• sufficient — usable for summarization

Do not summarize unless content_quality == "sufficient".

Prompt Injection Defense

• quarantined=true means the extractor detected instruction-like text in extracted content that was not present in the page's visible rendered text (common in .sr-only/visually-hidden abuse).
• When quarantined, content_quality is downgraded to minimal, policy_flags includes hidden_text_suspected and quarantined, and content/markdown outputs are blanked (fail-closed).

Error Format

{"error": "http_error|validation_error|internal_error", "status": 400, "details": {}}

Benchmarks

Combat arena — head-to-head benchmarks against Crawl4AI, Firecrawl (self-hosted), and Scrapy. All tests run on the same machine, same URLs, same conditions. Grub runs first as baseline, remaining adapters in randomized order with 10s delay between each to prevent rate-limiting bias.

Single-URL Speed (ms, lower is better)

Grub wins 4/5 single-URL speed races. Markdown conversion runs at 0-21ms via native Rust engine (grub_md).

Grub Phase Breakdown (server-side ms)

Navigation dominates; markdown conversion is sub-millisecond on most pages thanks to the Rust engine.

Batch Throughput (ms, lower is better)

Grub wins 2/3 batch sizes. Per-URL cost: 163-312ms (Grub) vs 255-477ms (others).

How to Run

# Start Grub
docker compose up -d

# Start Firecrawl (optional)
docker compose -f combat/firecrawl-compose.yaml up -d

# Install combat deps
pip install crawl4ai scrapy markdownify tabulate

# Run the arena
pytest combat/ -m combat -v

# Generate report
python -m combat.report

Development Status

Phase 1: Core Infrastructure ✅

Phase 2: Crawling ✅

Phase 3: Agent Module ✅

• Agent core — state machine, types, errors (W1)
• Unified tool contract — dispatcher with timeout/retry (W2)
• Policy gates — domain allowlist, private-range deny, redaction (W3)
• Observability — EventBus, TraceCollector, RunSummary persistence (W4)
• API wiring — /api/agent/run, /api/agent/status, JobType.AGENT_RUN (W5)
• Provider adapters — OpenAI, Anthropic, Ollama with fallback (W6)
• Config flags — agent, provider, ghost, stream settings (W7)

Phase 4: Ghost Protocol ✅

• Cloak-mode trigger detection (W8)
• Screenshot capture pipeline (W8)
• Vision extraction via Claude/GPT-4o (W8)
• Fallback chain in engine (W8)
• Ghost tool for external callers (W8)
• Ghost MCP tool + REST endpoint (W8)

Phase 5: Live Browser Stream ✅

• Persistent browser pool with lease/return (W9)
• CDP screencast relay (W9)
• WebSocket endpoint with interactive commands (W9)
• MJPEG fallback stream (W9)
• Stream status + pool status endpoints (W9)

Phase 5.5: Anti-Detection ✅

• Camoufox anti-detect browser engine (W10)
• Per-request proxy with env fallback (W10)
• Stealth patches for Chromium (W10)
• Tracker/analytics domain blocking (W10)
• Anthropic vision format detection fix (W10)

Phase 6: Mesh Coordinator ✅

• Peer discovery with gossip (1-hop) (W11)
• HMAC-SHA256 inter-node auth (W11)
• Heartbeat loop with load metrics + seed retry (W11)
• MeshDispatcher — transparent cross-node tool routing (W12)
• Load-based scoring with locality/affinity bonus (W12)
• Deploy scripts — local, mesh, Cloud Run (W12)
• Docker Compose 2-node mesh topology (W12)
• Embedded landing page (grub-site) (W12)

Phase 7: Performance + Hardening

• Rust markdown engine (grub_md) — PyO3 native extension, sub-ms conversion
• Combat arena — automated benchmarks vs Crawl4AI, Firecrawl, Scrapy
• Unit test suite — 176 tests across all modules
• Error handling improvements
• Monitoring and alerting

See MASTER_PLAN.md for the full architecture plan.

License

Grub Crawler Project License