Crawler

searxng/searxng

Privacy-respecting metasearch engine that aggregates results from multiple search services without tracking or profiling users. Self-hostable with…

koala73/worldmonitor

Real-time global intelligence dashboard. AI-powered news aggregation, geopolitical monitoring, and infrastructure tracking in a unified situational…

panniantong/agent-reach

Give your AI agent eyes to see the entire internet. Read & search Twitter, Reddit, YouTube, GitHub, Bilibili, XiaoHongShu — one CLI, zero API fees.

sqlmapproject/sqlmap

Automatic SQL injection and database takeover tool

microsoft/playwright

Browser automation framework with a single API for Chromium, Firefox, and WebKit. Supports web testing, scraping, screenshots, and network…

d4vinci/scrapling

Adaptive web scraping framework with anti-bot bypass, automatic element relocation, concurrent crawling, proxy rotation, and browser automation for…

naibowang/easyspider

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

projectdiscovery/nuclei

Fast YAML-based vulnerability scanner with template-driven detection engine for automated security testing across web apps, APIs, networks, DNS, and…

soxoj/maigret

🕵️‍♂️ Collect a dossier on a person by username from 6K websites

dgtlmoon/changedetection.io

Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price…

scrapegraphai/scrapegraph-ai

Python scraper based on AI

qeeqbox/social-analyzer

API, CLI, and Web App for analyzing and finding a person's profile in 1000 social media \ websites

zaproxy/zaproxy

Open-source web application security scanner for automated vulnerability detection, manual penetration testing, and API security testing with a…

skyvern-ai/skyvern

Automate browser based workflows with AI

projectdiscovery/nuclei-templates

Community curated list of templates for the nuclei engine to find security vulnerabilities.

ffuf/ffuf

Fast web fuzzer written in Go

maurosoria/dirsearch

Web path scanner

s0md3v/photon

Incredibly fast crawler designed for OSINT.