pentest-ai

It doesn't flag. It proves.

Website · Install · Why verification · Benchmarks · Limits · Discord

⚠️ Offensive tooling, authorized testing only. By installing you accept the AUP and Terms. See Responsible use ↓

Two minutes, no API key, no target of your own

pip install ptai && ptai demo

ptai demo scans a bundled vulnerable app and prints 4 findings, 3 oracle-VERIFIED. It replays one live from a proof capsule (replay 3/3), then runs the same routes hardened and prints 0 findings.

Two things to notice. The findings appear and disappear with the vulnerability rather than because the tool went quiet — the only thing that changed between the two runs is the fix. And one of the four stays a candidate: the SQLi login bypass is real, but no oracle could re-prove it on that route, so it does not get a badge. That gap is the product working, not a bug in the demo.

What VERIFIED actually means here

Most scanners tell you a thing might be exploitable and leave the triage to you. ptai treats a finding as a candidate until a named machine oracle re-runs the exploit and reproduces it N out of N times. Only then does it earn VERIFIED.

Three properties make that more than a slogan:

No LLM ever produces a verdict. The rule is enforced in code, not by policy: a verdict that cannot name the oracle that earned it is rejected. An LLM coordinates the run and reasons about results. It never decides whether a bug is real.

Every oracle has a control that must fail. A trusted-header bypass has to return privileged content with the header and a denial without it. A leaked-credential check has to be accepted for the real secret and rejected for a deliberately corrupted twin. An endpoint that answers 200 to everything earns nothing. This is what stops "it returned 200" from being mistaken for proof.

Third-party scanner output is held back. nuclei, nikto and zap results do not become findings on their own authority. They stay unverified until one of ptai's own oracles re-proves them independently.

Each VERIFIED finding ships as a portable proof capsule — the finding, the recipe to re-prove it, and the receipt. Anyone can ptai replay it against the live target and watch the oracle re-confirm, without trusting ptai. Capsules are deliberately unsigned: replay is the trust mechanism, not a signature you have to take on faith.

Honest numbers

On a deliberately vulnerable honeypot, 23 findings verify across those 14 classes at 100% precision with zero false positives. On a stock OWASP Juice Shop, 12 verify in a single scan.

Read those numbers carefully, because the gaps are the point. 63 probes exist but only 28 can earn a verdict; the other 35 report honest candidates. 203 wrappers are registered but only 18 turn tool output into findings — the rest run and hand back raw text. The oracle gate buys precision, not catch rate: it removes false positives, it does not find more bugs.

The honeypot harness (tests/honeypot/) and a clean-app zero-false-positive gate (tests/cleanapp/) both ship in this repo and run in CI, so these are reproducible rather than screenshots.

What it does not do

Stated plainly, because a security tool that oversells itself is worse than useless.

• It is a web application scanner. All 63 probes and all 23 oracle kinds target HTTP. AD, cloud, mobile and wireless have agents and tool wrappers, but no probe library and no oracles behind them.
• It cannot do local privilege escalation. That needs code execution on a host you already own. ptai tests remotely and has no such channel, so the privesc agent reports unsupported rather than a misleading zero.
• It is not a CVE scanner. There is no version-to-CVE database and no exploit library. CVE work is limited to osv.dev lookups on leaked manifests.
• Playbooks plan, they do not execute. ptai playbook run resolves dependencies and prints the plan. Running it against a target is not wired up yet.
• It is not autonomous. Fully autonomous LLM pentest agents finish 21–31% of tasks end to end; human-assisted setups reach 64%. ptai is built for the second regime. Press Ctrl+C twice to take over mid-run.

The complete internal defect list, including everything above, is tracked openly rather than quietly. If something here is wrong, open an issue and it gets corrected.

Install

pip install ptai
ptai mcp install          # auto-detects your MCP clients and writes their configs

pip install ptai
export ANTHROPIC_API_KEY=sk-...        # or OPENAI_API_KEY
ptai start https://target.example.com

# fully local, no cloud:
export PENTEST_AI_LLM_PROVIDER=ollama

# or deterministic, no LLM at all:
ptai start https://target.example.com --no-llm

ptai tools install --tier core     # or recommended / full
ptai tools install nmap nuclei     # or by name
ptai serve                         # HTTP REST + WebSocket for dashboards
ptai menu                          # interactive launcher, no LLM

Benchmarks

Reproducible, in git, with raw artifacts. No "98.7% detection rate" you can't audit.

n=1, single-rater, single-shot on OWASP Juice Shop. Methodology and raw output in benchmarks/; full write-up in docs/benchmarks/juice-shop.md.

The honest read: ptai is strong on SPA web targets with curated probe coverage. HexStrike is broader (cloud, binary, CTF) and likely beats ptai on traditional crawlable surfaces like WordPress. Juice Shop is also the most-documented vulnerable app on the internet, so both the LLM and the probe authors have a head start — which is exactly why the private honeypot number is lower, and why both get published.

Drop it into CI

- run: pip install ptai
- run: ptai start ${{ vars.STAGING_URL }} --ci --fail-on verified --sarif pentest.sarif
- uses: github/codeql-action/upload-sarif@v3
  with: { sarif_file: pentest.sarif }

--fail-on verified breaks the build only on a finding an oracle actually proved, so the gate cannot be tripped by scanner noise. SARIF uploads to GitHub Code Scanning, findings post as a PR comment. GitLab and Jenkins templates in docs/ci-cd.md.

How it works

recon ──▶ auth ──▶ web ──┬──▶ ad
                         ├──▶ cloud          ┌──────────────────┐
                         └──▶ api ──────────▶│  findings DB     │
                                             │  scope-guarded   │
                                             └────────┬─────────┘
                                                      ▼
                                          verify (oracles, N/N)
                                                      ▼
                                   chain ─▶ validate ─▶ detect ─▶ report
                                          md · html · pdf · SARIF · JUnit

18 specialist agents run the phases. With an API key each uses an LLM to reason about results; without one it runs as a deterministic tool loop. The phase order and the detection are identical either way — the probes find the bugs, the LLM only coordinates.

Who it's for

AppSec teams wiring an authenticated scan into every PR, with a gate that only trips on proven findings. Consultants who want the report to write itself and a capsule the client's own engineers can replay. Bug bounty hunters who'd rather triage 12 proven findings than 600 maybes. Claude Code / Cursor / Codex users who want real tooling behind their assistant without another API bill.

Work with me

The tool is MIT and free forever — that will not change.

If you want a pentest delivered rather than run yourself, or a hosted workspace with history and team access, both are at pentestai.xyz. Every finding in a delivered engagement ships with a proof capsule your engineers can replay themselves, which is a materially different artifact from a PDF full of severity ratings.

Questions: [email protected]

Responsible use

ptai executes real network and host operations against the targets you specify. You are solely responsible for having explicit written authorization for every target. Testing systems you do not own may violate the Computer Fraud and Abuse Act, the Computer Misuse Act 1990, GDPR Article 32, and equivalents elsewhere.

First run prompts for AUP acceptance and persists it. Set PENTEST_AI_AUP_ACCEPTED=1 in CI. Out-of-scope hosts are refused at tool-invocation time. Three guardrails ship off by default and are worth turning on: intensity=safe skips state-mutating probes, respect_rate_limits honours 429/Retry-After, and strict_scope refuses off-host requests.

ptai start http://target --oast-server https://oast.example.com --oast-token <T>
ptai start http://target --no-oast          # or disable entirely

FAQ

Do I need an API key? Not on the MCP path — your Claude Code / Cursor / Codex subscription is the LLM. Only the standalone CLI needs one, and even there Ollama runs fully local.

Is it autonomous? No, and it doesn't claim to be. The probes detect, the LLM coordinates, you decide. Ctrl+C twice takes over mid-run.

Safe against production? Only with written authorization and the three guardrails above turned on.

Does it phone home? No telemetry; findings stay on your disk. OAST callbacks are the one exception — see above, or run --no-oast.

How is this different from asking Claude to hack something? A curated deterministic probe library finds the bugs and a machine oracle proves them. An LLM alone gives you a plausible guess with no way to tell whether it's real.

Ecosystem

Community: Discord · Discussions · Issues

Star history

License

MIT. Do whatever you want with it.