Web crawling, scraping, spidering, and automated reconnaissance tools for security testing.
0xdsm/pinkerton🕵️ Python project to crawl for JavaScript files and search for secrets like API keys, authorization tokens, hardcoded credentials, etc.
menkrep1337/xssconXSSCon: Simple XSS Scanner tool
si9int/cc.pyExtracting URLs of a specific target based on the results of "commoncrawl.org"
hahwul/urxExtracts URLs from OSINT Archives for Security Insights
mhmdiaa/second-orderCrawls web applications to detect second-order subdomain takeover vulnerabilities by collecting URLs and matching configurable rules for non-200…
a3h1nt/grawlerGrawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them…
nox-project/nox-frameworkHigh-performance OSINT/CTI framework for automated identity pivoting and risk analysis across 120+ sources.
timf34/pagesourceCLI to download websites' actual JS/CSS/assets (not flattened HTML)
redhuntlabs/httplootAn automated tool which can simultaneously crawl, fill forms, trigger error/debug pages and "loot" secrets out of the client-facing code of sites.
c6fc/warcannonHigh speed/Low cost CommonCrawl RegExp in Node.js
khashashin/ogiOpen Source Link Analysis & OSINT Framework
iamstoxe/urlgrabA golang utility to spider through a website searching for additional links.
mrcl0wnlab/scanner-inurlbrAdvanced search in search engines, enables analysis provided to exploit GET / POST capturing emails & urls, with an internal custom validation…
dragonked2/egyscanEgyscan The Best web vulnerability scanner; it's a multifaceted security powerhouse designed to fortify your web applications against malicious…
joshkar/x-reconA utility for detecting webpage inputs and conducting XSS scans.