Crawler

tomnomnom/waybackurls

Fetch all the URLs that the Wayback Machine knows about for a domain

circl/ail-framework

AIL framework - Analysis Information Leak framework. Project moved to https://github.com/ail-project

ginlix-ai/langalpha

Automated OSINT framework for reconnaissance, data harvesting, and threat intelligence gathering with modular crawlers, scanners, and extraction…

wapiti-scanner/wapiti

Web vulnerability scanner written in Python3

thewhiteh4t/finalrecon

Automated web reconnaissance tool providing header analysis, DNS enumeration, subdomain discovery, directory scanning, SSL inspection, and port…

al0ne/vxscan

python3写的综合扫描工具，主要用来存活验证，敏感文件探测(目录扫描/js泄露接口/html注释泄露)，WAF/CDN识别，端口扫描，指纹/服务识别，操作系统识别，POC扫描，SQL注入，绕过CDN，查询旁站等功能，主要用来甲方自测或乙方授权测试，请勿用来搞破坏。

rndinfosecguy/scavenger

Crawler (Bot) searching for credential leaks on paste sites.

eldraco/domain_analyzer

Analyze the security of any domain by finding all the information possible. Made in python.

qianlitp/crawlergo

A powerful browser crawler for web vulnerability scanners

1tayh/noisy

Simple random DNS, HTTP/S internet traffic noise generator

uphiago/recon-skills

Field-validated offensive security skill pack with 169 techniques for reconnaissance and penetration testing. Covers CORS, SSRF, subdomain takeover,…

megadose/onionsearch

OnionSearch is a script that scrapes urls on different .onion search engines.

paulpierre/informer

A Telegram Mass Surveillance Bot in Python

mandatoryprogrammer/thermoptic

A next-generation HTTP stealth proxy which perfectly cloaks requests as the Chrome browser across all layers of the stack.

xnl-h4ck3r/waymore

Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, VirusTotal, GhostArchive & Intelligence X!

jaeles-project/gospider

Gospider - Fast web spider written in Go