Skip to content
KitploitKITPLOIT
ToolsExploitsBlog
Log in
Submit
ToolsExploitsBlog
Submit

Hacking, PenTest, and Cybersecurity Tools for Your Security Arsenal!

Kitploit is a directory of hacking, cybersecurity, and pentesting tools. Discover the latest project updates to find vulnerabilities, analyze systems, automate testing, and strengthen your security.

FeedsContactPrivacy© 2026 Kitploit

Tool Directory

Categories

View all categories
Loading categories
grubcrawler — The world's fastest agentic crawler. Reclaimed. Reinvented. Ready for war. | Kitploit
Tools/GitHubGitHub/deepbluedynamics/grubcrawler
OSINT (Open Source Intelligence)ReconnaissanceDynamic Analysis (Sandboxing)Web Proxies & InterceptionInformation GatheringWeb SecurityPenetration TestingUtilities & FrameworksMachine LearningRed TeamingCrawlerAnti-Bot
347571 month agoReviewed by Kitploit
Share
GitHubdeepbluedynamics/grubcrawler

grubcrawler

The world's fastest agentic crawler. Reclaimed. Reinvented. Ready for war.

View RepositoryWebsite
Grub Crawler

License Python FastAPI Playwright MCP Ghost Protocol Live Stream Camoufox Proxy


Agentic web crawler with anti-detection, vision fallback, and peer-to-peer mesh.



Full-stack web crawling engine with JavaScript rendering, Camoufox anti-detect browser, per-request proxy routing, and autonomous agent loops. Converts pages to clean markdown with a native Rust extraction engine. When standard crawling is blocked by Cloudflare, CAPTCHAs, or JavaScript walls, Ghost Protocol captures a screenshot and extracts content via vision AI (Claude, GPT-4o, or Ollama). Supports multi-provider LLM orchestration across OpenAI, Anthropic, and Ollama in a single session. Nodes coordinate over a gossip-based peer-to-peer mesh for distributed crawling.


An open source project from DeepBlue Dynamics, which builds open source agentic tooling for the marine electronics market. DeepBlue's promise: agents and people can ask questions of a boat's instruments, logs and documents, on board and without a connection.

Run it

# Full image: Camoufox (stealth Firefox) + Chromium. linux/amd64 and linux/arm64.
docker run -d --name grub -p 6792:6792 -e DISABLE_AUTH=true deepbluedynamics/grubcrawler:latest

# Lite image: Chromium only, about half the size. Raspberry Pi 5 and other small hosts.
docker run -d --name grub -p 6792:6792 -e DISABLE_AUTH=true deepbluedynamics/grubcrawler:latest-lite

curl http://localhost:6792/health

DISABLE_AUTH=true is for your own machine. Anywhere else, leave it unset and put the service behind nuts-auth tokens (Authorization: Bearer ahp_…), which is how the hosted instance at https://grub.nuts.services works.

Add it to your agent (MCP)

grub is an MCP server (streamable HTTP) at /mcp/ exposing grub_crawl, grub_screenshot and grub_diagnose. Point your harness at http://localhost:6792/mcp/ for your own container, or at https://grub.nuts.services/mcp/ with a bearer token for the hosted one.

Claude Code

claude mcp add --transport http grub http://localhost:6792/mcp/
# hosted
claude mcp add --transport http --header "Authorization: Bearer ahp_yourtoken" grub https://grub.nuts.services/mcp/

Cursor — .cursor/mcp.json

{ "mcpServers": { "grub": { "url": "http://localhost:6792/mcp/",
                          "headers": { "Authorization": "Bearer ahp_yourtoken" } } } }

VS Code (Copilot) — .vscode/mcp.json

{ "servers": { "grub": { "type": "http", "url": "http://localhost:6792/mcp/",
                       "headers": { "Authorization": "Bearer ahp_yourtoken" } } } }

Claude Desktop — claude_desktop_config.json, through mcp-remote

{ "mcpServers": { "grub": { "command": "npx",
    "args": ["-y", "mcp-remote", "https://grub.nuts.services/mcp/", "--header", "Authorization: Bearer ahp_yourtoken"] } } }

Codex CLI — ~/.codex/config.toml

[mcp_servers.grub]
url = "https://grub.nuts.services/mcp/"
bearer_token_env_var = "GRUB_TOKEN"   # export GRUB_TOKEN=ahp_yourtoken

Gemini CLI — ~/.gemini/settings.json

{ "mcpServers": { "grub": { "httpUrl": "http://localhost:6792/mcp/",
                          "headers": { "Authorization": "Bearer ahp_yourtoken" } } } }

The HTTP API has more surface than the MCP tools (batch, PDF pages, jobs, agent runs); see API Endpoints. Scanned-PDF OCR needs a vision provider: Ollama on the host (benhaotang/Nanonets-OCR-s, the docker-compose default) or an ANTHROPIC_API_KEY / OPENAI_API_KEY.

First crawl over HTTP

curl -X POST http://localhost:6792/api/markdown \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "customer_id": "me"}'

# a PDF comes back as markdown, one "## Page N" section per page
curl -X POST http://localhost:6792/api/markdown \
  -H "Content-Type: application/json" \
  -d '{"url": "https://arxiv.org/pdf/1706.03762", "customer_id": "me"}'

Images

TagBrowsersSize, unpacked (compressed pull)Platforms
deepbluedynamics/grubcrawler:latest, :<version>Camoufox (Firefox) + Playwright/Patchright Chromium~5 GB (2.9 GB)linux/amd64, linux/arm64
deepbluedynamics/grubcrawler:latest-lite, :<version>-litePlaywright/Patchright Chromium only~2.4 GB amd64, ~2.8 GB arm64 (1.2 GB)linux/amd64, linux/arm64

Both share one codebase and API. The lite image sets BROWSER_ENGINE=chromium; the full image adds the anti-detect Firefox engine, Xvfb and Ghost Protocol's stronger stealth. Build either yourself with docker build . or docker build -f Dockerfile.lite .; the release workflow publishes both for both architectures on every version tag.


Why Grub

We integrated features from every major crawler — then added what none of them have.

Self-Hosted Crawlers

FeatureCrawl4AIFirecrawlScrapyGrub
JS rendering✅ Playwright✅ Playwright❌ HTTP only✅ Playwright
Anti-detect browserstealth plugin❌❌✅ Camoufox
Ghost Protocol❌❌❌✅ auto fallback
Per-request proxy✅ escalation❌middleware✅ per-request
Stealth patches✅❌❌✅ opt-in
Agent loop✅ agentic✅ /agent❌ spiders✅ bounded SM
Live browser stream✅ WebSocket✅ Live View❌✅ WS + MJPEG
Markdown output✅ Fit Markdown✅ core❌✅ Rust engine
PDF extraction✅ PDF strategy✅ parse❌✅ text layer + OCR fallback
MCP tools✅ community✅ official⚠️ community✅ 15 tools
Multi-provider LLM✅ all LLMs⚠️ Gemini❌✅ OpenAI/Anthropic/Ollama
Mesh P2P❌❌❌✅ gossip protocol
Policy enforcement❌❌❌✅ domain gates + redaction
Prompt injection defense❌❌❌✅ quarantine + visible-text diff
LicenseApache 2.0AGPL-3.0BSDBSD-3-Clause
PricingFreeFree–$333/moFreeSelf-hosted

Cloud / Managed Crawlers

Download Tool

Most Popular

View all →

Discover the most used tools by our community.

  • 1

    OsintHunter

    letox4/osinthunter

    🎯 The ultimate OSINT toolkit: investigation graph, confidence scoring, alias families, map, timeline and reports. 49 modules, 400+ sites, no API keys required.

    524
  • 2

    ghostlock-app

    yukonga/ghostlock-app

    GhostLock One-Tap Execution App (CVE-2026-43499)

    478
  • 3

    ARTEX

    baarai/artex

    LLM multi-agent autonomous penetration testing platform with planner/worker agents, asset and exploration graphs, MITM traffic recording, and vulnerability retesting.

    357
  • 4

    redforge

    cybersecurity91/redforge

    Browser-side red team toolkit with 102 offline tools for recon, payload generation, credential attacks, crypto, encoding and engagement reporting.

    317
  • 5

    estorides

    grisuno/estorides

    Aggregates 99+ free OSINT sources in parallel, resolves entities into a knowledge graph, and exports STIX 2.1, MISP, or GraphML with LLM-assisted analysis.

    305
  • 6

    opencti

    opencti-platform/opencti

    Open Cyber Threat Intelligence Platform

    275

Explore all tools

Browse our collection of tools

View all tools →