Skip to content
KitploitKITPLOIT
उपकरणएक्सप्लॉइटब्लॉग
Log in
जमा करें
उपकरणएक्सप्लॉइटब्लॉग
जमा करें

हैकिंग, पेनटेस्ट और साइबर सुरक्षा उपकरण आपके सुरक्षा शस्त्रागार के लिए!

Kitploit हैकिंग, साइबर सुरक्षा और पेंटेस्टिंग टूल्स की एक निर्देशिका है। कमजोरियों को खोजने, सिस्टम का विश्लेषण करने, परीक्षण को स्वचालित करने और अपनी सुरक्षा को मजबूत करने के लिए नवीनतम प्रोजेक्ट अपडेट खोजें।

फ़ीडसंपर्कगोपनीयता© 2026 Kitploit

टूल निर्देशिका

श्रेणियाँ

सभी श्रेणियाँ देखें
Loading categories
उपकरण/GitHubGitHub/germondai/trawl
Web Proxies & InterceptionInformation GatheringWeb SecurityUtilities & FrameworksCrawlerAnti-BotFingerprint SpoofingCAPTCHA Bypass
GitHubgermondai/trawl

trawl

Self-hosted scraping engine — bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.

रिपॉजिटरी देखें
1.2k677353 दिन पहलेKitploit द्वारा समीक्षित
साझा करें
वेबसाइट
अनुरोधित भाषा में सामग्री उपलब्ध नहीं है। अंग्रेज़ी संस्करण दिखाया जा रहा है।
<h1 align="center">
  <a href="https://trawl.germondai.com" target="_blank">
    <img align="center" src="https://assets.kitploit.com/production/public/readmes/placeholders/f0fc86cfe65f76d40e15aaec61704ec8220a56dc89d4be03c46f67cb31b9fa8c.svg" /><br/><br/>
    <span>TRAWL</span>
  </a>
</h1>

Self-hosted web scraping for applications and AI agents. Get page content through an API or MCP, with built-in handling for supported browser challenges and CAPTCHAs.

TRAWL starts with HTTP, reuses verified clearance and opens Camoufox when a browser is needed. Repeat requests fetch fresh content without repeating a full solve whenever the site accepts the saved session.

`HTTP + supported PoW walls → cached browser session → fresh browser solve → residential proxy`

The HTTP tier can clear supported ALTCHA and Friendly Captcha v1 access forms with bounded local computation. Ordinary embedded widgets keep their browser lifecycle. [Coverage and limits](https://github.com/germondai/trawl/blob/dev/apps/docs/getting-started/challenge-support.md).

[Website](https://trawl.germondai.com) · [Documentation](https://docs.trawl.germondai.com) · [Latest release](https://github.com/germondai/trawl/releases/latest)

## Features

- **Challenge handling:** Cloudflare, Anubis, NCBI proof of work, Akamai, Imperva, DDoS-Guard and supported AWS WAF and DataDome flows. Built-in CAPTCHA paths include Turnstile, reCAPTCHA v2, hCaptcha, GeeTest Slide, ALTCHA, Friendly Captcha, CAP and supported Shumei spatial puzzles. [Coverage and limits](https://github.com/germondai/trawl/blob/dev/apps/docs/getting-started/challenge-support.md).
- **Fast repeat requests:** automatic HTTP clearance reuse, Redis or in-memory caching, and browser fallback when required. No named session is needed for shared clearance reuse.
- **MCP for agents:** read Markdown, scrape HTML, extract structured JSON, capture screenshots and inspect browser activity from known URLs.
- **Isolated browser sessions:** retain cookies, localStorage, sessionStorage and browser identity. Scrape within a session, keep a page open for JavaScript eval and read cookies. [Session API](https://github.com/germondai/trawl/blob/dev/apps/docs/api-reference/browser-sessions.md).
- **API and forward proxy:** use `/scrape`, the FlareSolverr-compatible `/v1` endpoint, or a challenge-aware HTTP/HTTPS proxy with WebSockets and Range support.
- **Local dashboard:** persistent request history, tier outcomes, failure causes and live updates.
- **Local solving by default:** optional 2Captcha fallback and opt-in Firefox TLS/HTTP2 impersonation; Bun fetch remains the default HTTP transport.

## Sponsors

<details open>
  <summary>View/Collapse All</summary>

  <table>
    <tr>
      <td width="30%" align="center" valign="middle">
        <a href="https://get.brightdata.com/trawl" target="_blank">
          <img width="1254" height="1254" alt="Birght Data" src="https://assets.kitploit.com/production/public/readmes/56362/da67b80ddc744ec86558bcf3581568d6e39af074df31d30aec0440a8a181e92f/8faf1303c90aa63bcc7f972e403d1cb568ed0bb303784e404f1c5c3f2b6e9b09-display-v1.webp" />
        </a>
      </td>
      <td valign="middle">
        <b><a href="https://get.brightdata.com/trawl" target="_blank">Bright Data</a></b> - The most powerful platform for Web Unlocker, SERP API and web scraping tools.<br><br>
        <b>Why <a href="https://get.brightdata.com/trawl" target="_blank">Bright Data</a>?</b><br>
        • <a href="https://get.brightdata.com/trawl-web-unlocker" target="_blank">Web Unlocker</a> - bypass any anti-bot protection<br>
        • <a href="https://get.brightdata.com/trawl-serp-api" target="_blank">SERP API</a> - real-time Google, Bing & more results<br>
        • Scraping Browser & dedicated scrapers<br>
        • Massive residential proxy network<br>
        • Built for scale and reliability<br><br>
        <b>Get started for free with <a href="https://get.brightdata.com/trawl" target="_blank">Bright Data</a>!</b>
      </td>
    </tr>
    <tr>
      <td width="30%" align="center" valign="middle">
        <a href="https://go.nodemaven.com/germondaioctoberGH" target="_blank">
          <img width="4096" height="1024" alt="NodeMaven" src="https://assets.kitploit.com/production/public/readmes/56362/5115ee0faf9b1eeb705ba380608d10ef95a0dd35050468ea9718a4b20e5fb074/dc64a6531bbfaf385f39356dc5e162d36053559cbdb230261f3d08c679efb706-display-v1.webp" />
        </a>
      </td>
      <td valign="middle">
        <b><a href="https://go.nodemaven.com/germondaioctoberGH" target="_blank">NodeMaven</a></b> - The most efficient proxy provider for Web Scrapping and Automation with the Highest Quality IP on the market.<br><br>
        <b>Why <a href="https://go.nodemaven.com/germondaioctoberGH" target="_blank">NodeMaven</a>?</b><br>
        • ZIP targeting<br>
        • 99.9% uptime<br>
        • IP filtering: all proxies have fraud score <97%<br>
        • No KYC required<br>
        • Unique free tools: Proxy Bandwidth Checker, Meta Tag Checker, IP Lookup and others!<br><br>
        <b>Special codes for <a href="https://trawl.germondai.com" target="_blank">TRAWL</a> users:</b><br>
        • <code>TRAWL35</code> - 35% off to Mobile and Residential Proxies<br>
        • <code>TRAWL40</code> - 40% off to ISP (Static) Proxies<br>
        • <code>TRAWL2X</code> - 2X traffic for Black Friday
      </td>
    </tr>
    <tr>
      <td width="30%" align="center" valign="middle">
        <a href="https://www.thordata.com/?ls=dtw&lk=dtw" target="_blank">
          <img width="1254" height="1254" alt="Thordata" src="https://assets.kitploit.com/production/public/readmes/56362/58bb0faef9771e5a7d802b1f836f622679713de70b9523958bb983b6c82c6b15/bc29c4196c2a099c04bd62522f8f75c5c7a973ac8eedc28a94156c7ca537fd14-display-v1.webp" />
        </a>
      </td>
      <td valign="middle">
        <b><a href="https://www.thordata.com/?ls=dtw&lk=dtw" target="_blank">Thordata</a></b> - Residential Proxies & Web Data APIs<br><br>
        <p>Give <b>TRAWL</b> reliable upstream access with <b>100M+ residential IPs across 190+ countries</b>, including city and ASN targeting.</p>
        <b>Why <a href="https://www.thordata.com/?ls=dtw&lk=dtw" target="_blank">Thordata</a>?</b><br>
        • <b>Residential Proxies:</b> rotating and sticky sessions for scraping and browser automation<br>
        • <b>SERP API:</b> structured Google, Bing, and other search results for research workflows<br>
        • <b>Web Scraper API:</b> collect public web data through a simple API<br><br>
        <b><a href="https://trawl.germondai.com" target="_blank">TRAWL</a> users:</b><br>
        • Start with free trial traffic and get <b>10% off</b> with code <code>TRAWL10</code>
      </td>
    </tr>
  </table>
</details>

## Performance

Selected same-machine comparisons show faster first and repeat responses than FlareSolverr and Byparr on the default Cloudflare workload:

| Protected page          | Best first | Best repeat |
| ----------------------- | ---------: | ----------: |
| NopeCHA Cloudflare demo |  **6.0 s** |  **879 ms** |
| 1337x                   |  **3.8 s** |   **85 ms** |
| EZTV                    |  **4.6 s** |  **473 ms** |

Measured **2026-10-17** on a development build after v1.8.0. These are verified minima from the recorded sample. First requests used a warm browser with an empty target cache; repeats are fresh fetches with reusable clearance. Results vary by site, IP and session state.

[Full comparison, medians and CPU/RAM](https://github.com/germondai/trawl/blob/dev/benchmarks/RESULTS.md) · [Method](https://github.com/germondai/trawl/blob/dev/benchmarks/README.md) · [Performance guide](https://github.com/germondai/trawl/blob/dev/apps/docs/getting-started/performance.md) · [Low-resource setup](https://github.com/germondai/trawl/blob/dev/apps/docs/getting-started/low-resource.md)

## Quick start

Run one browser with MCP, an in-memory clearance cache and persistent dashboard history:

```bash
docker run -d --name trawl \
  -p 127.0.0.1:8191:8191 \
  --shm-size=1gb \
  -e SESSION_CACHE_DRIVER=memory \
  -e BROWSER_POOL_SIZE=1 \
  -e MCP_ENABLED=true \
  -e METRICS_DASHBOARD_ENABLED=true \
  -v trawl_metrics:/data/metrics \
  ghcr.io/germondai/trawl:1.11.1

curl http://localhost:8191/health
```

Open [the local dashboard](http://localhost:8191/dashboard). Keep the API and MCP endpoint on a trusted network; dashboard history survives container recreation through the mounted volume.

For Redis and Compose deployments, see the [quick-start guide](https://github.com/germondai/trawl/blob/dev/apps/docs/getting-started/quick-start.md) and [Compose examples](https://github.com/germondai/trawl/blob/dev/apps/docs/deployment/docker-compose.md). Use `:1.11.1-baseline` on older CPUs or NAS hardware.

<details>
<summary>Dashboard preview</summary>

![TRAWL metrics dashboard showing illustrative request data](https://assets.kitploit.com/production/public/readmes/56362/9f7f53f4924b8ca585338ea4466a7978a16575d770c0ef0db65ec691e3c2163a/9ade488990fb5b196d342d28b8226f233175e73ae63ed2646d3c9f5c900db7fb-display-v1.webp)

The screenshot uses illustrative traffic. Your dashboard shows requests actually handled by your instance. [Metrics setup](https://github.com/germondai/trawl/blob/dev/apps/docs/api-reference/health-stats.md#local-metrics-dashboard).

</details>

## API

### Native scraping

```bash
curl -X POST http://localhost:8191/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://nopecha.com/demo/cloudflare","solveEmbeddedCaptchas":false}'
```

Returns HTML, cookies, the winning tier and timings. Set `solveEmbeddedCaptchas: false` for page content only; omit it to attempt supported embedded form widgets. Access challenges still need to clear. A delivered page does not by itself prove a form CAPTCHA was solved.

[Native API](https://github.com/germondai/trawl/blob/dev/apps/docs/api-reference/native-api.md) · [Custom headers](https://github.com/germondai/trawl/blob/dev/apps/docs/api-reference/custom-headers.md)

### JavaScript and TypeScript client

The `packages/client` workspace provides a typed client for Node.js, Bun, Deno and
browser applications, including scrape options, session handles and structured errors.
Build it locally before the first npm publication. [Client guide](https://github.com/germondai/trawl/blob/dev/packages/client/README.md).

### Prowlarr, Jackett and FlareSolverr clients

```bash
curl -X POST http://localhost:8191/v1 \
  -H 'Content-Type: application/json' \
  -d '{"cmd":"request.get","url":"https://nopecha.com/demo/cloudflare"}'
```

Set your client's FlareSolverr URL to `http://trawl:8191` on a shared Docker network, or the TRAWL host address otherwise. `/v1` prioritizes page delivery and skips optional embedded widget solving by default.

[Prowlarr setup](https://github.com/germondai/trawl/blob/dev/apps/docs/integrations/prowlarr.md) · [Jackett setup](https://github.com/germondai/trawl/blob/dev/apps/docs/integrations/jackett.md) · [Compatibility reference](https://github.com/germondai/trawl/blob/dev/apps/docs/api-reference/flaresolvr-compat.md)

### MCP for AI clients

With `MCP_ENABLED=true`, connect your client using **Streamable HTTP** at `http://localhost:8191/mcp` or the TRAWL host address. Tools: `read`, `scrape`, `extract`, `screenshot` and `inspect`, plus the `scrape_url` compatibility alias.

TRAWL loads known public URLs; it does not provide web search. Reuse browser sessions with `sessionId`; create sessions used by MCP with `publicOnly: true`. [MCP guide](https://github.com/germondai/trawl/blob/dev/apps/docs/integrations/mcp.md).

### Browser sessions

Create an isolated session and pass its ID on subsequent requests:

```bash
curl -X POST http://localhost:8191/sessions \
  -H 'Content-Type: application/json' \
  -d '{"id":"research"}'

curl -X POST http://localhost:8191/sessions/research/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://nopecha.com/demo/cloudflare","solveEmbeddedCaptchas":false}'

curl -X DELETE http://localhost:8191/sessions/research
```

Sessions preserve browser storage between requests and keep login state out of the shared clearance cache. Requests within one session are sequential. `/scrape` with `sessionId` remains equivalent to `/sessions/:id/scrape`.

By default, request pages close after each operation. Create a session with `"keepPage": true` to retain one main page and use `/sessions/:id/eval` to run JavaScript in it. Sessions expire when idle and do not survive an API restart.

[Session API, authentication and lifecycle](https://github.com/germondai/trawl/blob/dev/apps/docs/api-reference/browser-sessions.md)

## Challenge and CAPTCHA support

TRAWL detects supported walls and runs the matching browser flow before returning content. Coverage includes JavaScript and proof-of-work challenges, interactive controls, and supported CAPTCHA widget variants.

Some challenges need a particular browser mode, a different egress IP or an external provider. For example, DataDome Device Check uses the opt-in headful pool; built-in hCaptcha support does not include image grids. Passive reCAPTCHA v3 scripts are not treated as blocking widgets.

The optional 2Captcha fallback supports captured Cloudflare Challenge Pages and supported declarative Turnstile/reCAPTCHA v2 widgets without a site profile. Other documented tasks require a site profile. It is disabled without a provider key; paid-service acceptance remains unverified.

[Full coverage matrix and external task catalogue](https://github.com/germondai/trawl/blob/dev/apps/docs/getting-started/challenge-support.md) · [Solver configuration](https://github.com/germondai/trawl/blob/dev/apps/docs/getting-started/configuration.md#optional-external-captcha-solver)

## HTTP/HTTPS forward proxy

Enable `MITM_ENABLED=true` and point your HTTP-compatible client at port `8192`. Normal traffic is forwarded directly; detected challenge responses escalate through TRAWL's browser tiers. HTTPS clients must trust the generated TRAWL CA. Keep the proxy private and persist its CA directory.

[Proxy setup](https://github.com/germondai/trawl/blob/dev/apps/docs/proxy/overview.md) · [Install the CA](https://github.com/germondai/trawl/blob/dev/apps/docs/proxy/ca-installation.md) · [Prowlarr and other clients](https://github.com/germondai/trawl/blob/dev/apps/docs/proxy/client-setup.md)

## Docker images

| Tags                             | Use case                             |
| -------------------------------- | ------------------------------------ |
| `:latest`, `:1.11.1`             | Standard release for modern hardware |
| `:baseline`, `:1.11.1-baseline`   | Older x86 CPUs and NAS kernels       |
| `:nightly`, `:nightly-<dev-sha>`  | Development builds from `dev`        |

All tags use `ghcr.io/germondai/trawl`. Both release variants are published for Linux AMD64 and ARM64. [Standalone containers and source builds](https://github.com/germondai/trawl/blob/dev/apps/docs/deployment/standalone.md).

<details>
<summary>Compose variants and NAS app catalogs</summary>

- [docker-compose.yml](https://github.com/germondai/trawl/blob/dev/docker-compose.yml): scraper and Redis.
- [docker-compose.minimal.yml](https://github.com/germondai/trawl/blob/dev/docker-compose.minimal.yml): scraper with in-memory caching.
- [docker-compose.prod.yml](https://github.com/germondai/trawl/blob/dev/docker-compose.prod.yml): Redis, restart policy and health checks.
- [TrueNAS Community Apps](https://apps.truenas.com/catalog/trawl_community/) and [Unraid Community Apps](https://ca.unraid.net/apps/trawl-1o4q23p06utr4h): community-maintained packages.

</details>

## Configuration

| Setting                               | Default | Purpose                                                   |
| ------------------------------------- | ------- | --------------------------------------------------------- |
| `BROWSER_POOL_SIZE`                   | `1`     | Warm browsers available for concurrent work               |
| `SESSION_CACHE_DRIVER`                | `redis` | Choose Redis or single-process `memory` caching           |
| `SCRAPE_MAX_TIMEOUT_MS`               | `60000` | Default scrape deadline; request timeouts take precedence |
| `TIER1_TRANSPORT`                     | `bun`   | Set `wreq` for optional Firefox TLS/HTTP2 impersonation   |
| `BROWSER_SESSION_MAX_ENTRIES`         | `4`     | Maximum named browser sessions per instance               |
| `PROXY_URL` / `RESIDENTIAL_PROXY_URL` | unset   | Optional upstream proxies and proxy pools                 |

[All settings](https://github.com/germondai/trawl/blob/dev/apps/docs/getting-started/configuration.md) · [Environment example](https://github.com/germondai/trawl/blob/dev/.env.example) · [Upgrade guide](https://github.com/germondai/trawl/blob/dev/apps/docs/deployment/configuration-migration.md)

## Documentation and development

- [Architecture and tier routing](https://github.com/germondai/trawl/blob/dev/apps/docs/architecture/tiered-execution.md)
- [Clearance cache and HTTP reuse](https://github.com/germondai/trawl/blob/dev/apps/docs/architecture/session-cache.md)
- [Browser pool and resource management](https://github.com/germondai/trawl/blob/dev/apps/docs/architecture/browser-pool.md)
- [Troubleshooting](https://github.com/germondai/trawl/blob/dev/apps/docs/deployment/troubleshooting.md)
- [Release history](https://github.com/germondai/trawl/blob/dev/CHANGELOG.md)
- [Contributing and local development](https://github.com/germondai/trawl/blob/dev/CONTRIBUTING.md)

<details>
<summary>Monorepo structure and stack</summary>

| Workspace          | Purpose                                            |
| ------------------ | -------------------------------------------------- |
| `apps/api`         | Bun + Elysia API, MCP, dashboard and forward proxy |
| `apps/web`         | Nuxt landing page                                  |
| `apps/docs`        | VitePress documentation                            |
| `packages/browser` | Camoufox pool and clearance cache                  |
| `packages/tiers`   | Routing, browser sessions and challenge solvers    |
| `packages/types`   | Shared types                                       |

</details>

## Star History

<a href="https://www.star-history.com/?repos=germondai%2Ftrawl">
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=germondai/trawl&type=date&theme=dark&legend=top-left" />
   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=germondai/trawl&type=date&legend=top-left" />
   <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=germondai/trawl&amp;type=date&amp;legend=top-left" />
 </picture>
</a>

## Contributors

<a href="https://github.com/germondai/trawl/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=germondai/trawl" />
</a>

---

<p align="center">
    <span>Made with ❤️ by</span>
    <a href="https://github.com/germondai" target="_blank">@germondai</a>
</p>
टूल डाउनलोड करें

सबसे लोकप्रिय

सभी देखें →

हमारे समुदाय द्वारा सबसे अधिक उपयोग किए जाने वाले उपकरण खोजें।

  • 1

    OsintHunter

    letox4/osinthunter

    🎯 अंतिम OSINT टूलकिट: जाँच ग्राफ़, विश्वास स्कोरिंग, उपनाम परिवार, मानचित्र, टाइमलाइन और रिपोर्ट। 49 मॉड्यूल, 400+ साइटें, किसी API कुंजी की आवश्यकता नहीं।

    732
  • 2

    ghostlock-app

    yukonga/ghostlock-app

    # घोस्टलॉक वन-टैप निष्पादन ऐप (CVE-2026-43499)

    506
  • 3

    ARTEX

    baarai/artex

    प्लानर/वर्कर एजेंट, एसेट और एक्सप्लोरेशन ग्राफ़, MITM ट्रैफ़िक रिकॉर्डिंग, और भेद्यता पुनःपरीक्षण के साथ LLM मल्टी-एजेंट स्वायत्त पेनेट्रेशन टेस्टिंग प्लेटफ़ॉर्म।

    405
  • 4

    redforge

    cybersecurity91/redforge

    ब्राउज़र-साइड रेड टीम टूलकिट, जिसमें रेकॉन, पेलोड जनरेशन, क्रेडेंशियल हमलों, क्रिप्टो, एन्कोडिंग और एंगेजमेंट रिपोर्टिंग के लिए 102 ऑफ़लाइन टूल्स हैं।

    342
  • 5

    CyberStrikeAI

    ed1s0nz/cyberstrikeai

    AI-मूल साइबरसुरक्षा के लिए कार्रवाई की प्रणाली—जहाँ इरादा नियंत्रित निष्पादन बन जाता है, साक्ष्य परिचालन स्मृति बन जाता है, और हर ऑपरेशन अगले को बेहतर बनाता है।

    261
  • 6

    antifox

    antifox-ai/antifox

    AI मल्टी-एजेंट स्वायत्त पेनेट्रेशन टेस्टिंग प्लेटफ़ॉर्म जो रेकॉन, अटैक-सरफ़ेस डिस्कवरी, एक्सप्लॉइटेशन वैलिडेशन और साक्ष्य-आधारित रिपोर्ट डिलीवरी संचालित करता है।

    259

सभी उपकरण खोजें

हमारे उपकरणों का संग्रह ब्राउज़ करें

सभी उपकरण देखें →