Crawler

dermotblair/webvulscan

Web Application Vulnerability Scanner.

dwisiswant0/galer

A fast tool to fetch URLs from HTML attributes by crawl-in.

k3ystr0k3r/intrack

A flexible internet crawler used for scanning technologies, instances and vulnerabilities worldwide across the internet.

caio-ishikawa/netscout

OSINT tool that finds domains, subdomains, directories, endpoints and files for a given seed URL.

ksharinarayanan/sourcewolf

Amazingly fast response crawler to find juicy stuff in the source code! 😎🔥

0xzdh/bridgekeeper

Scrape, Hunt, and Transform names and usernames

s0md3v/predator

Anti-Automation System

h33tlit/jbin-website-secret-scraper

Jbin will gather all the URLs from the website and then it will try to expose the secret data from them such as API keys, API secrets, API tokens and…

morph3/crawpy

Asynchronous HTTP content discovery tool with auto-calibration, recursive scanning, and report generation for enumerating hidden web directories and…

soheilkhodayari/jaw

JAW: A Graph-based Security Analysis Framework for Client-side JavaScript

reedgraff/hingesdk

Hinge Dating App SDK for python for lonely bored SWEs

urbanadventurer/bing-ip2hosts

bingip2hosts is a Bing.com web scraper that discovers websites by IP address

them4hd1/pencrawler

An Advanced Web Crawler and DirBuster

fetchfox/fetchfox

AI-powered web scraping and data extraction library that extracts structured information from any webpage using natural language instructions, with…

hehpollon/instagram-crawler

Non API. Crawling post (photo, likes, comments, date ...) by username, hashtags

kacakb/jsfinder

Fetches JavaScript files quickly and comprehensively.