Crawler

yogeshojha/rengine

reNgine is an automated reconnaissance framework for web applications with a focus on highly configurable streamlined recon process via Engines,…

firecrawl/firecrawl-mcp-server

MCP server providing web scraping, crawling, and search capabilities for AI agents. Supports structured data extraction, interactive browsing, and…

urbanadventurer/whatweb

Next generation web scanner

mishakorzik/allhackingtools

All-in-One Hacking Tools For Hackers! And more hacking tools! For termux.

constverum/proxybroker

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS :performing_arts:

kaliiiiiiiiii-vinyzu/patchright

Undetected version of the Playwright testing and automation library.

arachni/arachni

Web Application Security Scanner Framework

jasonxtn/argus

The Ultimate Information Gathering Toolkit

tuhinshubhra/red_hawk

All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers

edoardottt/cariddi

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

seladb/pcapplusplus

Multiplatform C++ library for high-performance network packet capture, parsing, crafting, and analysis. Supports libpcap, DPDK, AF_XDP, PF_RING, and…

qianlitp/crawlergo

A powerful browser crawler for web vulnerability scanners

jaeles-project/gospider

Gospider - Fast web spider written in Go

xnl-h4ck3r/waymore

Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, VirusTotal, GhostArchive & Intelligence X!

spider-rs/spider

Get web data for AI agents and LLMs

apurvsinghgautam/dark-web-osint-tools

OSINT Tools for the Dark Web

anouarbensaad/vulnx

vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform…