Distributed Systems Testing Skills

Two skills for AI coding agents that design and run claim-driven tests for distributed and stateful systems. Together they produce a structured Markdown test plan and a findings report with 10-state verdicts and an explicit SUT / harness / checker / environment blame classification. A reviewer reads the two artifacts and decides whether to ship; nothing else has to be re-run.

Works with Claude Code, Codex, Copilot CLI, Cursor, Gemini, or any agent that reads Markdown and runs shell. The skills are plain SKILL.md files. The agent executes them; the plan and findings report are the output.

One skill designs the plan. The other runs it. A plan starts from the product's claims, generates hypotheses tied to those claims, and writes scenarios named after the claim each tries to falsify. For consistency-critical scenarios, each scenario also binds an abstract model (register | queue | log | lock | lease | ledger | …) to an operation-history schema, a named checker, and a nemesis with observable landing evidence. The plan ends with a coverage adequacy argument and a conservative confidence statement.

Why

The default for testing distributed and stateful systems — write a few integration tests and call it done — finds a small fraction of the bugs that actually break these systems in production: partial network partitions, non-deterministic concurrency, crash-recovery, upgrade/rollback, idempotency under replay, timing-sensitive ordering.

These skills enforce an opinionated workflow that pulls from the field's hard-won knowledge:

• Claim-driven, not test-driven. Start from what the product promises. Every scenario falsifies one claim under one fault. A test named after its claim is harder to weaken than one named after its setup.
• Coverage adequacy is a deliverable. The plan ends with an argument that the chosen scenarios are enough to ship, plus an honest list of what stays unverified.
• Reuse the SUT's own toolbox. The execute skill discovers existing tests, runbooks, and fault-injection scaffolding before inventing anything new.
• Model + history + checker, not just chaos. For safety, durability, idempotency, isolation, ordering, or membership claims, every scenario declares an abstract model, an operation-history schema, a named checker (linearizability, serializability, session-consistency, no-lost-ack, exactly-once, …), and how it treats ambiguous outcomes (timeouts, unknown commits, retries). Chaos plus a model and a checker, not chaos alone.
• No silent passes. Every PASS cites oracle execution evidence and the signal proving the fault actually fired. Verdicts come from a 10-state set, so "the chaos script ran cleanly" can't be read as "the claim survived the fault." Every FAIL carries a SUT / harness / checker / environment blame tag so reproducers reach the right queue.

What you get

End-to-end, the two skills produce:

docs/testing-plans/<slug>.md        ← plan with §0–§9 (see below)
test-sessions/<slug>/<UTC>/
  ├── session-log.md                 ← timeline + toolbox + env probe
  ├── logs/                          ← per-scenario stdout/stderr
  ├── metrics/                       ← metric snapshots
  ├── artifacts/                     ← ephemeral harnesses, dumps
  └── findings/
      ├── <scenario>.md              ← per-scenario verdict (written as run proceeds)
      └── report.md                  ← summary + adequacy + confidence delta

The plan structure (a reviewer can read this and decide whether to ship without re-running the tests):

0. Architectural summary       — system as it actually exists
1. Scope
1b. Claims under test          — the spine
1c. Missing claims discovered  — docs ↔ code drift
2. SUT model
3. Existing test inventory     — what's already covered
4. Failure-mode hypotheses     — tied to claim IDs
5. Coverage matrix             — claim × hypothesis
6. Technique selection         — from the catalog
6b. Environment requirements
7. Scenarios                   — each named after the claim, with
                                  Target test file + Skeleton
   7.M Model / history /       — mandatory when the scenario falsifies
       checker discipline        a claim in {safety, durability,
                                  idempotency, isolation, ordering,
                                  membership}: model under test,
                                  operation-history schema, named
                                  checker, nemesis + landing evidence,
                                  ambiguous-outcome handling, reduction
                                  plan (SUT/harness/checker/env blame)
7b. Coverage adequacy argument — why these tests are enough
7c. Residual uncertainty       — what stays unverified, and why ok
7d. Confidence statement       — the reviewer's verdict
8. What this plan does NOT cover
9. Open questions / followups

Example §7.M block (excerpt from a plan)

### Scenario S3: linearizable_append_under_partition
- Falsifies if it FAILs: C1 (every acknowledged append is durable
  and linearisable), C5 (leader election completes within 5s)
- Workload: 8 clients, 70% append / 30% read, 5min, key-skew zipf
- Faults: asymmetric partition isolating current leader at T+60s
  for 30s
- Oracle: linearizability via Porcupine over per-key histories

§7.M (model / history / checker discipline)
- Model under test:    log
- Operation history:   default 11-field schema (op id, process id,
                       invoke/complete ts, op type, key, input,
                       output, error, timeout marker, node seen,
                       fault epoch). Recorded in-process + server-
                       side audit.
- Checker:             linearizability (Porcupine) per-key, then
                       no-lost-ack against final state
- Nemesis + landing:   asymmetric-partition (iptables drop one
                       direction). Landing evidence = iptables drop
                       counter goes 0 → 14,712 over the 30s window
                       AND raft log emits "leader-lost; starting
                       election" within 2s of injection.
- Ambiguous outcomes:  timeouts → timeout_marker=true, complete_ts
                       =null, treated as could-have-succeeded;
                       retries are separate ops sharing input
- Reduction plan:      if FAIL, bisect fault window + fix seed, then
                       classify SUT / harness / checker / environment
                       per references/test-case-reduction.md

Example findings-report row

ID	Verdict	Nemesis landing evidence	Reduction class
S3	PASS-hardening	iptables ctr 0→14,712; raft re-election at T+1.8s	n/a
S4	FAIL-reproducible	partition landed; Elle: G2-item anomaly on key K17	SUT
S7	INCONCLUSIVE-fault-not-proven	iptables rule installed but counter stayed 0 — wrong chain	harness
S9	PARTIAL-model	landing ok; checker covered per-key, not cross-key	n/a

(The full findings template carries Oracle, Oracle execution evidence, artifact links, an adequacy-vs-plan section, and a confidence delta — see skills/executing-distributed-system-tests/assets/findings-report-template.md.)

Install (one line, any agent)

Paste this at any AI coding agent (Claude Code, Codex, Copilot CLI, Cursor, Gemini, or anything else that reads Markdown and runs shell):

Read https://raw.githubusercontent.com/shenli/distributed-system-testing/main/INSTALL.md
and follow the instructions to install and configure
distributed-testing-skills for this agent.