
Multi-threaded web crawler for structured site mapping and regex-based data extraction. Exports results in JSON, XML, or TXT for reconnaissance and OSINT workflows.
π Crawl websites efficiently, extract structured data, and visualize connections. π΅οΈββοΈ
π³ Structured Web-Tree Generation
π΅οΈ Regex-Based Data Extraction
β‘ High-Speed Multi-threading
π Multiple Export Formats
π¨ Colorized Output & Robust Error Handling
git clone https://github.com/XORbit01/webpalm.git
cd webpalm
go build -o webpalm && ./webpalm
go install github.com/XORbit01/webpalm/v2@latest
webpalm -h
π -i, --include # Include only specific domains (e.g., google.com, facebook.com)
π -u, --url # Target website
π -l, --level # Depth of traversal
β -x, --exclude # Exclude status codes (e.g., 404, 500)
πΎ -o, --output # Save results (JSON, XML, TXT)
π -w, --worker # Multi-threading workers
π --regexes # Extract data using regex
webpalm -u https://example.com -l2
webpalm -u https://example.com -l1 --regexes comments="\<\!--.*?-->" -o results.json
webpalm -u https://example.com -l3 -w 50
webpalm -u https://example.com -l2 -o output.xml
| π Purpose | π Regex Pattern |
|---|---|
| π§ Emails | [a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+ |
| π¬ Comments | \<\!--.*?--> |
| π Tokens | [a-zA-Z0-9]{32} |
| π Passwords | \bpassword\b.{0,10} |
π Escape special characters if needed.
π‘ Pull requests are welcome! Open an issue before major changes.
π’ Discord: xorbit.