grubcrawler

An open source project from DeepBlue Dynamics, which builds open source agentic tooling for the marine electronics market. DeepBlue's promise: agents and people can ask questions of a boat's instruments, logs and documents, on board and without a connection.

Run it

# Full image: Camoufox (stealth Firefox) + Chromium. linux/amd64 and linux/arm64.
docker run -d --name grub -p 6792:6792 -e DISABLE_AUTH=true deepbluedynamics/grubcrawler:latest

# Lite image: Chromium only, about half the size. Raspberry Pi 5 and other small hosts.
docker run -d --name grub -p 6792:6792 -e DISABLE_AUTH=true deepbluedynamics/grubcrawler:latest-lite

curl http://localhost:6792/health

DISABLE_AUTH=true is for your own machine. Anywhere else, leave it unset and put the service behind nuts-auth tokens (Authorization: Bearer ahp_…), which is how the hosted instance at https://grub.nuts.services works.

Add it to your agent (MCP)

grub is an MCP server (streamable HTTP) at /mcp/ exposing grub_crawl, grub_screenshot and grub_diagnose. Point your harness at http://localhost:6792/mcp/ for your own container, or at https://grub.nuts.services/mcp/ with a bearer token for the hosted one.

Claude Code

claude mcp add --transport http grub http://localhost:6792/mcp/
# hosted
claude mcp add --transport http --header "Authorization: Bearer ahp_yourtoken" grub https://grub.nuts.services/mcp/

Cursor — .cursor/mcp.json

{ "mcpServers": { "grub": { "url": "http://localhost:6792/mcp/",
                          "headers": { "Authorization": "Bearer ahp_yourtoken" } } } }

VS Code (Copilot) — .vscode/mcp.json

{ "servers": { "grub": { "type": "http", "url": "http://localhost:6792/mcp/",
                       "headers": { "Authorization": "Bearer ahp_yourtoken" } } } }

Claude Desktop — claude_desktop_config.json, through mcp-remote

{ "mcpServers": { "grub": { "command": "npx",
    "args": ["-y", "mcp-remote", "https://grub.nuts.services/mcp/", "--header", "Authorization: Bearer ahp_yourtoken"] } } }

Codex CLI — ~/.codex/config.toml

[mcp_servers.grub]
url = "https://grub.nuts.services/mcp/"
bearer_token_env_var = "GRUB_TOKEN"   # export GRUB_TOKEN=ahp_yourtoken

Gemini CLI — ~/.gemini/settings.json

{ "mcpServers": { "grub": { "httpUrl": "http://localhost:6792/mcp/",
                          "headers": { "Authorization": "Bearer ahp_yourtoken" } } } }

The HTTP API has more surface than the MCP tools (batch, PDF pages, jobs, agent runs); see API Endpoints. Scanned-PDF OCR needs a vision provider: Ollama on the host (benhaotang/Nanonets-OCR-s, the docker-compose default) or an ANTHROPIC_API_KEY / OPENAI_API_KEY.

First crawl over HTTP

curl -X POST http://localhost:6792/api/markdown \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "customer_id": "me"}'

# a PDF comes back as markdown, one "## Page N" section per page
curl -X POST http://localhost:6792/api/markdown \
  -H "Content-Type: application/json" \
  -d '{"url": "https://arxiv.org/pdf/1706.03762", "customer_id": "me"}'

Images

Tag	Browsers	Size, unpacked (compressed pull)	Platforms
deepbluedynamics/grubcrawler:latest, :<version>	Camoufox (Firefox) + Playwright/Patchright Chromium	~5 GB (2.9 GB)	linux/amd64, linux/arm64
deepbluedynamics/grubcrawler:latest-lite, :<version>-lite	Playwright/Patchright Chromium only	~2.4 GB amd64, ~2.8 GB arm64 (1.2 GB)	linux/amd64, linux/arm64

Both share one codebase and API. The lite image sets BROWSER_ENGINE=chromium; the full image adds the anti-detect Firefox engine, Xvfb and Ghost Protocol's stronger stealth. Build either yourself with docker build . or docker build -f Dockerfile.lite .; the release workflow publishes both for both architectures on every version tag.

---

Why Grub

We integrated features from every major crawler — then added what none of them have.

Self-Hosted Crawlers

Feature	Crawl4AI	Firecrawl	Scrapy	Grub
JS rendering	✅ Playwright	✅ Playwright	❌ HTTP only	✅ Playwright
Anti-detect browser	stealth plugin	❌	❌	✅ Camoufox
Ghost Protocol	❌	❌	❌	✅ auto fallback
Per-request proxy	✅ escalation	❌	middleware	✅ per-request
Stealth patches	✅	❌	❌	✅ opt-in
Agent loop	✅ agentic	✅ /agent	❌ spiders	✅ bounded SM
Live browser stream	✅ WebSocket	✅ Live View	❌	✅ WS + MJPEG
Markdown output	✅ Fit Markdown	✅ core	❌	✅ Rust engine
PDF extraction	✅ PDF strategy	✅ parse	❌	✅ text layer + OCR fallback
MCP tools	✅ community	✅ official	⚠️ community	✅ 15 tools
Multi-provider LLM	✅ all LLMs	⚠️ Gemini	❌	✅ OpenAI/Anthropic/Ollama
Mesh P2P	❌	❌	❌	✅ gossip protocol
Policy enforcement	❌	❌	❌	✅ domain gates + redaction
Prompt injection defense	❌	❌	❌	✅ quarantine + visible-text diff
License	Apache 2.0	AGPL-3.0	BSD	BSD-3-Clause
Pricing	Free	Free–$333/mo	Free	Self-hosted

Cloud / Managed Crawlers