ApiHarvester

A full-pipeline black-box API security scanner written in Python. It enumerates endpoints, identifies parameters, probes HTTP methods, tests authentication/authorization logic, and runs standard OWASP API Top 10 attack simulations (BOLA, Broken Auth, BFLA, Mass Assignment, Rate Limiting, SSRF, Security Misconfiguration, etc.), plus a RESTler-style reliability fuzzer that hunts crashes/500s independent of any OWASP category.

All core scripts are designed to be stdlib-only (standard library only). If external tools or libraries are not present on the system, apiharvester automatically falls back to pure-Python implementations to guarantee out-of-the-box execution.

---

Repository Structure

• apiharvester/ — The main python package directory. Run as python3 -m apiharvester.
• scripts/check_requirements.sh — Verifies binary + payload prerequisites.
• scripts/install_requirements.sh — Downloads required SecLists payload files and optionally installs binaries (via go install and pip).
• apiharvester.py — A standalone, single-file distribution of the scanner.
• api_deep_discovery.py — Dynamic crawler using Katana headless browser code for dynamic SPA/XHR endpoints discovery.
• api_intelligence_engine.py — Pipeline aggregator and passive vulnerability classifier.
• apisec.py — Alternate single-file security scanner version.
• requirements.txt — Python dependencies list (primarily for optional Python-based accelerators).
• payloads/ — Wordlist and payload files for reconnaissance:
  • params.txt — 25,889 parameter name candidates for API endpoint testing
  • directories.txt — 62,281 common API path patterns and directory names
  • subdomains.txt — 5,000 subdomain variants for API discovery
  • kiterunner/ — Kiterunner route schema files for accelerated endpoint enumeration

---

Installation & Setup

1. Verify Requirements: Run the read-only script to check if core tools/payloads are available:

   ./scripts/check_requirements.sh
   

2. Install Optional Tools & Payload Files: Run the install script to automatically fetch SecLists top wordlists, Kiterunner route schemas, and install tool accelerators:

   ./scripts/install_requirements.sh
   

3. Install Python Packages:

   pip3 install -r requirements.txt
   

---

Usage

Run the scanner directly against a target domain:

python3 -m apiharvester example.com \
    --auth "Bearer eyJ..." \
    --auth2 "Bearer eyJ_lowpriv..." \
    --threads 20 \
    --html report.html \
    --json findings.jsonl

Command-line Options:

• target (positional): FQDN domain to scan.
• --auth: High-privilege access token for authenticated checks (e.g., valid user session).
• --auth2: Low-privilege access token for BOLA / BFLA / cross-account privilege-escalation testing.
• --threads: Threadpool size (default: 20).
• --timeout: HTTP request timeout in seconds (default: 10).
• --burst: Rapid request count for rate-limiting verification (default: 20).
• --json: Save JSONL format report (line-delimited JSON findings).
• --html: Save interactive HTML dashboard report.
• --output-dir: Override default output directory path (e.g., ./scans/example.com).
• --skip-recon: Skip recon phases, use existing output files from a prior run.
• --recon-dir: Load pre-existing recon output directory and run only attack phases.
• --attacks-only: Run only attack phases (implies --skip-recon).
• --attacks: Comma-separated attack list to run. Default: all. Available:

  bola,broken_auth,mass_assignment,rate_limit,bfla,business_logic,
  ssrf,misconfiguration,inventory,sspp,injection,reliability,secrets
  

Attack Modules

OWASP API Top 10 (API1–API10:2023):

• API1: BOLA/IDOR (bola) — Broken Object-Level Authorization. Tests object-ID endpoints with ID fuzzing (0, 1, 2, 99, "admin", "test", UUID variants, etc.) and differential auth tokens.
• API2: Broken Auth (broken_auth) — Unauthenticated endpoint discovery, JWT weak-secret cracking, alg=none bypass, claim tampering, kid injection, plus OPTIONS/HEAD method bypasses.
• API3: Mass Assignment (mass_assignment) — Privilege-escalation field injection into PUT/PATCH bodies (role, is_admin, verified, balance, etc.).
• API4: Rate Limiting (rate_limit) — Sends 20+ rapid requests; flags endpoints returning 200 instead of 429 Retry-After.
• API5: BFLA (bfla) — Broken Function-Level Authorization. Tests sensitive paths (/admin, /roles, /impersonate, etc.) with and without low-priv token.
• API6: Business Logic (business_logic) — Workflow/state-machine violations (e.g., updating an order after payment).
• API7: SSRF (ssrf) — Tests for server-side request forgery via URL parameters and request bodies.
• API8: Misconfiguration (misconfiguration) — CORS (active: sends untrusted Origin), missing security headers, verbose errors, server banner leaks.
• API9: Inventory (inventory) — Undocumented endpoints, abandoned endpoints, exposed admin interfaces.
• API10: SSPP (sspp) — Unsafe Server-Side Post Processing (template injection, XPath injection, etc.).

Bonus Attacks:

• Injection (injection) — SQL injection, XSS, command injection (error-based + time-based blind).
• Reliability (reliability) — RESTler-style fuzzing: boundary/malformed-input testing to find 5xx crashes and server reliability bugs (independent of OWASP categories).
• Secrets (secrets) — Pattern matching for leaked credentials in response bodies: AWS Access Keys, Google API Keys, Slack Tokens, Stripe Keys, GitHub Tokens, Private Key Blocks, JWTs, and generic secret assignments (api_key=..., password=..., etc.).

Example Scans

Full scan with authenticated + low-priv token (best for BOLA/BFLA):

python3 -m apiharvester api.example.com \
    --auth "Bearer high_priv_token_here" \
    --auth2 "Bearer low_priv_token_here" \
    --html report.html \
    --json findings.jsonl

Quick recon-only (discover endpoints, no attacks):

python3 -m apiharvester example.com --skip-recon --attacks ""

(Or just don't provide --auth to skip some attack phases.)

Re-run only attacks against saved recon data (fast iteration):

python3 -m apiharvester example.com --recon-dir output/example.com_20260715_140233 --attacks-only

Run only specific attacks (e.g., BOLA + Secrets):

python3 -m apiharvester example.com --attacks bola,secrets

Bypass TLS certificate errors (corporate proxy, staging environment):

# apiharvester uses a permissive TLS context by default — no extra flags needed
# All HTTPS endpoints work even with self-signed/intercepted certs
python3 -m apiharvester https://staging-api.example.com

---

Output Directory

Every scan writes its recon artifacts to a structured output directory before the attack phase runs. By default this is:

output/{target}_{YYYYMMDD_HHMMSS}/

e.g. output/example.com_20260715_140233/. Override the location with --output-dir /path/to/dir if you want a fixed, predictable path (useful for scripting/CI).

Files written