odomojuli/regextokenslist of regex patterns for oauth / api tokens with provided source
A sourced, tested, offline-proving scanner for OAuth / API tokens and secrets. Every pattern is sourced (primary provider docs), tested (positive + negative samples), and proven (RE2-compatible, ReDoS-checked). Ships a CLI with confidence tiers. Source of truth is DATA in build_patterns.py; patterns.json and this file are generated from it.
109 patterns / 64 providers / generated 2026-07-02
pip install -e . # console script + importable package
regextokens scan PATH... # walk files/dirs, report findings
regextokens scan PATH -m verified # only offline-proven findings
regextokens scan PATH -f sarif # SARIF for GitHub code scanning
regextokens list # show the catalog
Exit status: 0 no findings, 1 findings at/above the requested tier, 2 error. Point it at a diff or tree in CI to gate merges. Secrets are redacted in output by default.
Accept reviewed findings (test fixtures, docs examples) so CI only fails on new secrets:
regextokens scan . --write-baseline .regextokens-baseline.json # snapshot current findings
regextokens scan . --baseline .regextokens-baseline.json # subtract them
The baseline stores fingerprints (sha256 of pattern id + hashed secret body), never the secrets themselves, so it is safe to commit. Matching is line-independent — editing above an accepted finding does not un-suppress it — but the same token in a different file is a new exposure and is reported. A hand-edited allow section holds policy: fnmatch path globs and pattern ids. --write-baseline preserves and applies it. This repo commits its own baseline covering the synthetic samples in build_patterns.py; CI self-scans against it.
# .pre-commit-config.yaml
repos:
- repo: https://github.com/odomojuli/regextokens
rev: v0.2.0
hooks:
- id: regextokens
# .github/workflows/secrets.yml
jobs:
secrets:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: odomojuli/regextokens@main
with:
min-confidence: probable
# baseline: .regextokens-baseline.json
# optional: feed the SARIF to code scanning
# - uses: github/codeql-action/upload-sarif@v3
# if: always()
# with: {sarif_file: regextokens.sarif}
Both default to -m probable: the low tier includes public identifiers and placeholders that would make a commit gate unusable.
Every finding is scored by offline proof — local computation only, no network, no calls to the issuer. A shape match answers "does this look like a token?"; these checks answer "can we prove, locally, that it is or isn't one?" Filter with -m/--min-confidence.
verified-offline — a checksum or decoder proves the structure is authentic (GitHub/npm CRC32; empirically validated against real revoked tokens).probable — structure plus a Shannon-entropy body consistent with a real secret.low — shape match only; may be a placeholder or public identifier.rejected — offline proof says this is provably not a real token (e.g. CRC32 mismatch); dropped before it reaches you.The rejected tier is the differentiator: a ghp_/npm_ string with a valid prefix and length but a bad checksum is dropped with certainty, killing the largest false-positive class for the highest-value providers. Offline proof never claims a key is live — that still needs the issuer's API.
pip install pytest
pytest # validate every pattern + engine
python build_patterns.py # regenerate patterns.json + README.md
Consume the catalog directly (no engine):
import json, re
pats = json.load(open("patterns.json"))["patterns"]
rx = {p["id"]: re.compile(p["regex"]) for p in pats}
rx["github-pat-classic"].search(text)
build_patterns.py — source of truth (DATA) + validator; rebuilds patterns.json and README.md.patterns.json — generated catalog (machine-readable). Never edit by hand.regextokens/ — installable package: catalog (load), scanner (walk/match), verify (offline proof), baseline (accepted findings + allowlist), report (human/JSON/SARIF), cli. Bundles a synced copy of patterns.json..pre-commit-hooks.yaml / action.yml / .github/workflows/ — distribution: pre-commit hook, composite GitHub Action, CI + tag-triggered PyPI release (trusted publishing).tests/ — test_patterns.py (every pattern: compile, match, RE2, ReDoS) and test_engine.py (scanner, verify, report, CLI, catalog sync).sniffer-audit.md — audit of other secret scanners (verification, staleness).references.bib / references.md — bibliography of the secret-detection literature.re. No lookbehind or backreferences.re.search (scanning embedded text). Wrap with ^/$ to use as validators.\b word boundaries bracket most tokens. Drop them if scanning inside base64 blobs.prefix = reliable alone; structure = fixed shape; keyword = gate on a nearby keyword; identifier = public, not secret; encoding = format check, not secret.patterns.json are synthetic filler, not live credentials.sniffer-audit.md — how major scanners build and verify patterns, with benchmark data.references.bib / references.md — bibliography of the secret-detection literature.Base64 string [encoding]
^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$
Format validator, not a secret. Standard Base64 alphabet with padding. Anchored.
src: https://datatracker.ietf.org/doc/html/rfc4648#section-4
JSON Web Token (JWT) [structure]
\bey[A-Za-z0-9_-]{10,}\.ey[A-Za-z0-9/_-]{10,}\.[A-Za-z0-9/_-]{10,}
Three base64url segments. Header begins ey (i.e. {"). Decode to inspect; signature alone is not the secret.
src: https://datatracker.ietf.org/doc/html/rfc7519
PEM private key block [prefix]
-----BEGIN (?:RSA |EC |DSA |OPENSSH |PGP |ENCRYPTED )?PRIVATE KEY-----
Highest-value single indicator. Matches the header line of any PEM-encoded private key.
src: https://datatracker.ietf.org/doc/html/rfc7468
age X25519 secret key [structure]
\bAGE-SECRET-KEY-1[QPZRY9X8GF2TVDW0S3JN54KHCE6MUA7L]{58}\b
Bech32 with HRP 'AGE-SECRET-KEY-'; a 32-byte key encodes to 58 chars after the '1' separator (uppercase). Recipients (age1...) are public, not secrets. Post-quantum variant uses HRP AGE-SECRET-KEY-PQ-.
src: https://github.com/C2SP/C2SP/blob/main/age.md
Auth token (DEPRECATED API) [structure]
\bamzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b