Crawler

megadose/onionsearch

OnionSearch is a script that scrapes urls on different .onion search engines.

a3h1nt/grawler

Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them…

dwisiswant0/galer

A fast tool to fetch URLs from HTML attributes by crawl-in.

chvancooten/bugbountyscanner

A Bash script and Docker image for Bug Bounty reconnaissance. Intended for headless use.

dc4ts/changetower

Lightweight web page change monitor written in Go. Detects updates on target URLs and sends notifications via Telegram or other services, ideal for…

c6fc/warcannon

High speed/Low cost CommonCrawl RegExp in Node.js

r3nt0n/bopscrk

Generates targeted password wordlists by combining personal info, applying case/leet transforms, and scraping artist lyrics for enhanced cracking…

qianlitp/crawlergo

A powerful browser crawler for web vulnerability scanners

dheerajmadhukar/karma_v2

⡷⠂𝚔𝚊𝚛𝚖𝚊 𝚟𝟸⠐⢾ is a Passive Open Source Intelligence (OSINT) Automated Reconnaissance (framework)

offensive-hub/black-widow

GUI based offensive penetration testing tool (Open Source)

xnl-h4ck3r/waymore

Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan, VirusTotal, GhostArchive & Intelligence X!

yevgen2020/vk-geosearch

CLI tool to search and scrape geotagged photos from VK.com using the VK API, extracting owner names and photo metadata into an HTML report.

hahwul/deadfinder

Fast CLI tool to scan websites, sitemaps, and URL lists for broken links, with concurrent workers, coverage reporting, and multiple output formats.

redhuntlabs/httploot

An automated tool which can simultaneously crawl, fill forms, trigger error/debug pages and "loot" secrets out of the client-facing code of sites.

xnl-h4ck3r/xnlinkfinder

A python tool used to discover endpoints, potential parameters, a target specific wordlist for a given target and secrets

bellingcat/octosuite

Terminal-based toolkit for GitHub data analysis.