docker run --rm ghcr.io/crissyfield/troll-a [flags] [url]

# Install dependencies
brew install re2

# Install with RE2 activated
go install -tags re2_cgo github.com/crissyfield/[email protected]

# Install dependencies
sudo apt install -u build-essential libre2-dev

# Install with RE2 activated
go install -tags re2_cgo github.com/crissyfield/[email protected]

Usage:
  troll-a [flags] [url]

This tool allows to extract (potential) secrets such as passwords, API keys, and tokens
from WARC (Web ARChive) files. Extracted information is output as structured text org
JSON, which simplifies further processing of the data.

"url" can be either a regular HTTP or HTTPS reference ("https://domain/path"), an Amazon
S3 reference ("s3://bucket/path"), a file path (either "file:///path" or simply "path"),
or a dash ("-") to read from STDIN. If "url" is omitted data is read from STDIN. If the
input data is compressed with either GZip, BZip2, XZ, or ZStd it is automatically
decompressed. ZStd with a prepended custom dictionary (as used by "*.megawarc.warc.zstd")
is also handled transparently.

This tool uses rules from the Gitleaks project (https://gitleaks.io) to detect secrets.

Flags:
  -c, --custom stringArray     additional custom rule to apply. Secrets that match the
                               given regular expression (using RE2 syntax) will also be
                               reported. Can be specified multiple times.
  -e, --enclosed               only report secrets that are enclosed within their context
  -f, --filter string          filter for the target URL of each WARC record. Only WARC
                               records that match the given regular expression (using RE2
                               syntax) will be checked for secrets. An empty filter will
                               match everything.
  -h, --help                   help for troll-a
  -j, --jobs uint              detect secrets with this many concurrent jobs (default 8)
  -s, --json                   output detected secrets as JSON
  -p, --preset rules-preset    rules preset to use. This could be one of the following:
                               all:         All known rules will be applied, which can
                                            result in a significant amount of noise for
                                            large data sets.
                               most:        Most of the rules are applied, skipping the
                                            biggest culprits for false positives.
                               secret:      Only rules are applied that are most likely
                                            to result in an actual leak of a secret.
                               none:        No rules at all are applied. This can be used
                                            in combination with custom rules via the
                                            --custom/-c switch.
                               No other values are allowed. (default secret)
  -q, --quiet                  suppress success message(s)
  -r, --retry retry-strategy   retry strategy to use. This could be one of the following:
                               never:       This strategy will fail after the first fetch
                                            failure and will not attempt to retry.
                               constant:    This strategy will attempt to retry up to 5
                                            times, with a 5s delay after each attempt.
                               exponential: This strategy will attempt to retry for 15
                                            minutes, with an exponentially increasing
                                            delay after each attempt.
                               always:      This strategy will attempt to retry forever,
                                            with no delay at all after each attempt.
                               No other values are allowed. (default never)
  -t, --timeout duration       fetching timeout (does not apply to files) (default 30m0s)
  -v, --version                version for troll-a

# Download the list of all 90.000 WARC paths
curl -sSL -O https://data.commoncrawl.org/crawl-data/CC-MAIN-2023-50/warc.paths.gz

# Iterate through all paths using 64 scanning jobs, output matches as JSON
gzcat warc.paths.gz | \
xargs -I{} -- troll-a -e -s -j64 https://data.commoncrawl.org/{} > secrets.json

# Call troll-a directly with the MegaWARC URL
troll-a -e https://archive.org/download/archiveteam_pastebin_20230421003309_a3b951b4/pastebin_20230421003309_a3b951b4.1603050931.megawarc.warc.zst

Detected: secret="acf30fb56amsh654fa8104418601p1e420cjsn3152a0032f0b" rule="rapidapi-access-token" uri="https://pastebin.com/raw/bKMJXkQE" line=36 column=15
Detected: secret="acf30fb56amsh654fa8104418601p1e420cjsn3152a0032f0b" rule="rapidapi-access-token" uri="https://pastebin.com/raw/bKMJXkQE" line=36 column=15
Detected: secret="acf30fb56amsh654fa8104418601p1e420cjsn3152a0032f0b" rule="rapidapi-access-token" uri="https://pastebin.com/raw/nferefe2" line=37 column=6
Detected: secret="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" rule="github-pat" uri="https://pastebin.com/print/cQEA2GCS" line=39 column=123
Detected: secret="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" rule="github-pat" uri="https://pastebin.com/embed_js/cQEA2GCS" line=11 column=2688
Detected: secret="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" rule="github-pat" uri="https://pastebin.com/embed_iframe/cQEA2GCS?theme=dark" line=49 column=123
Detected: secret="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" rule="github-pat" uri="https://pastebin.com/cQEA2GCS" line=222 column=123
Detected: secret="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" rule="github-pat" uri="https://pastebin.com/raw/cQEA2GCS" line=22 column=22
Detected: secret="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" rule="github-pat" uri="https://pastebin.com/embed_iframe/cQEA2GCS" line=48 column=123
Detected: secret="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" rule="github-pat" uri="https://pastebin.com/embed_js/cQEA2GCS?theme=dark" line=11 column=2796
Detected: secret="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" rule="github-pat" uri="https://pastebin.com/clone/cQEA2GCS" line=152 column=27
Success: Processed https://archive.org/download/archiveteam_pastebin_20230421003309_a3b951b4/pastebin_20230421003309_a3b951b4.1603050931.megawarc.warc.zst (113372 records)