🕷️ SCRAPPER by BertUI

The Session Observer for Web Automation

SCRAPPER isn't a scraper — it's a SESSION OBSERVER that captures your real browser data for use in any automation tool.
Built on the BertUI React Framework
GitHub: BunElysiaReact/SCRAPY
No domain. No cloud. All local. All yours.

---

📋 Table of Contents

• The Problem SCRAPPER Solves
• What SCRAPPER Is (And Isn't)
• How SCRAPPER Works
• What SCRAPPER Captures
• Advantages & Disadvantages
• Universal Data API
• Quick Start — Linux / macOS
• Quick Start — Windows
• Browser Extensions
• Using Captured Data in Your Tools
• Dashboard Overview
• Production Use & Automation
• Contributing

---

🤔 The Problem SCRAPPER Solves

Every web automation tool — Puppeteer, Playwright, Selenium, even curl — shares the same challenges:

Challenge	Why It's Hard
Authentication	Manually scripting logins for every site is tedious and fragile
Session state	Cookies expire, tokens rotate, localStorage gets cleared
Reverse engineering	Hours spent in DevTools understanding API patterns
Bot detection	TLS fingerprints, browser entropy, Cloudflare, hCaptcha
Setup complexity	Fighting with headless browsers, proxies, and stealth plugins

The real issue: All these tools are trying to imitate a human. But they're guessing at what a real human looks like.

---

💡 What SCRAPPER Is (And Isn't)

SCRAPPER IS...	SCRAPPER IS NOT...
🔍 A session observer that watches YOUR real browser	❌ A replacement for Puppeteer/Playwright/Selenium
💾 A data capture tool that saves your actual session	❌ A tool that scrapes websites for you
📡 A local API server serving your captured data	❌ A hosted service or cloud platform
🧠 A reverse engineering assistant revealing hidden APIs	❌ A magic "scrape anything" button
🎯 A visual debugger for understanding site structure	❌ A no-code automation builder

SCRAPPER doesn't scrape. It gives you the REAL data YOU need to scrape successfully.

---

🔄 How SCRAPPER Works

Phase 1: Capture (Browser Open, You Browse)

YOU                                              SCRAPPER
  │                                                  │
  ├── Open Brave/Chrome/Firefox with extension ──────►│
  │                                                  │
  ├── Log into sites you want to automate ───────────►│ captures:
  │                                                  │  • Cookies
  ├── Browse normally, click buttons ────────────────►│  • Tokens
  │                                                  │  • Fingerprint
  └── Done browsing ─────────────────────────────────►│  • API requests
                                                     │  • DOM structure

Phase 2: Automate (Browser Can Close, You Code)

YOUR SCRIPT ──── GET /api/v1/session/all ────► SCRAPPER API (localhost:8080)
              ◄── { cookies, tokens, fingerprint } ────────────────────────┘
                │
                ▼
         Puppeteer / Playwright / Selenium / Python requests / curl
                │
                ▼
         ✅ Authenticated requests with YOUR real session

---

🎬 See It In Action

The Dashboard — 258 captured claude.ai requests

The Extension Popup — Live feed, quick capture, real-time stats

SCRAPPER Extension

The Installer — One command, 5 steps, done

Register Extension ID — One command after loading extension

Register Extension

Real World Example — Talking to Claude.ai from the terminal via SCRAPPER

The script above used SCRAPPER to capture a real claude.ai session, then sent messages directly via the API — zero login code, zero Puppeteer, zero browser automation.

---

📦 What SCRAPPER Captures

📦 Session Data
   ├── 🍪 Cookies (including HttpOnly, Secure, all domains)
   ├── 💾 localStorage & sessionStorage
   ├── 🔑 Auth tokens (Bearer, JWT, CSRF, custom)
   └── 📨 All HTTP headers

🖥️ Browser Fingerprint
   ├── 📱 User Agent
   ├── 🖼️ Screen resolution & color depth
   ├── 🌍 Timezone & language settings
   └── 📨 Full header set (Accept, Accept-Language, etc.)

📡 Network Traffic
   ├── 📤 All HTTP requests (URLs, methods, headers, POST data)
   ├── 📥 All HTTP responses (status, headers, bodies)
   └── 🔄 WebSocket frames

🌳 DOM State
   ├── 📄 DOM snapshots
   ├── 🎯 Live selector testing
   └── 🗺️ DOM maps (all tags, classes, IDs)

---

⚖️ Advantages & Disadvantages

✅ Advantages

Advantage	Why It Matters
Bypasses Advanced Bot Detection	Uses curl_cffi to impersonate a real browser's TLS fingerprint (e.g., Chrome 120) — not flagged as automated
97% Success Rate	Targets internal API routes, not visual UI — immune to CSS changes, moving buttons, or layout updates
Low Resource Usage	~20MB RAM vs 500MB+ for Puppeteer/Selenium. No browser engine running
Invisible Authentication	Piggybacks off your existing human-verified session — no login flow, no CAPTCHAs
Syncs With Real Browser	Messages/actions from scripts appear in your real browser tab when you refresh
Language Agnostic	Session API works with Python, Go, Rust, Node, curl — anything that can make HTTP requests

❌ Disadvantages

Disadvantage	What It Means
Brittle Session Lifespan	Entirely dependent on an active browser session — expires if you log out
Depends on Internal APIs	Uses undocumented endpoints that can change without notice
Requires Setup Infrastructure	Not standalone — needs native host + browser extension running simultaneously
Account Risk	Uses your real identity. Aggressive rate-limit hitting can get your real account banned
Learning Curve	Must read network traffic to understand correct API payloads
Single Device Binding	device-id is tied to one captured session — can't easily share across machines

---

📡 Universal Data API

Once captured, SCRAPPER serves everything via a simple REST API at http://localhost:8080.

Core Endpoints

Endpoint	Description
GET /api/v1/session/cookies?domain=example.com	All cookies for a domain
GET /api/v1/session/localstorage?domain=example.com	localStorage data
GET /api/v1/session/all	Complete session dump
GET /api/v1/fingerprint	Browser fingerprint
GET /api/v1/tokens/all	All extracted tokens
GET /api/v1/requests/recent?limit=50	Recent network requests
GET /api/v1/dom/snapshot?url=example.com	DOM snapshot
GET /api/v1/export/env	Environment variables format
GET /api/v1/bulk/all?format=[json|jsonl|har|csv|txt]	Everything, your format

---

🚀 Quick Start — Linux / macOS

One-Line Install