Crawler

redhuntlabs/httploot

An automated tool which can simultaneously crawl, fill forms, trigger error/debug pages and "loot" secrets out of the client-facing code of sites.

mhmdiaa/second-order

Crawls web applications to detect second-order subdomain takeover vulnerabilities by collecting URLs and matching configurable rules for non-200…

sethsec/celerystalk

An asynchronous enumeration & vulnerability scanner. Run all the tools on all the hosts.

andreiverse/gorilla

tool for generating wordlists or extending an existing one using mutations.

xorbit01/webpalm

Multi-threaded web crawler for structured site mapping and regex-based data extraction. Exports results in JSON, XML, or TXT for reconnaissance and…

hisxo/jspector

A simple Burp Suite extension to crawl JavaScript (JS) files in passive mode and display the results directly on the issues

digitalmethodsinitiative/dmi-tcat

Digital Methods Initiative - Twitter Capture and Analysis Toolset

timf34/pagesource

CLI to download websites' actual JS/CSS/assets (not flattened HTML)

justrach/kuri

Browser automation, web crawling, and iOS + Android device control for AI agents. Zig-native, token-efficient CDP snapshots, HAR recording, native…

iamstoxe/urlgrab

A golang utility to spider through a website searching for additional links.

nox-project/nox-framework

High-performance OSINT/CTI framework for automated identity pivoting and risk analysis across 120+ sources.

bernsteining/instaloctrack

An Instagram OSINT tool to collect all the geotagged locations available on an Instagram profile in order to plot them on a map, and dump them in a…

captaingeech42/ransomwatch

Ransomware leak site monitoring

edoverflow/megplus

Automated reconnaissance wrapper — TomNomNom's meg on steroids. [DEPRECATED]

dragonked2/egyscan

Egyscan The Best web vulnerability scanner; it's a multifaceted security powerhouse designed to fortify your web applications against malicious…