dobybaxter127/nimbus-vestigeA forensic reconstruction engine for cloud and identity incident response.
# Nimbus Vestige (NV)
**A forensic reconstruction engine for cloud and identity incident response.**
Given fragmented cloud, SaaS, and identity telemetry — control-plane logs, sign-in
events, token and consent activity — NV reconstructs how an intrusion most likely
moved across accounts and services, enumerates the other most-likely paths it could
have taken, and reports every step with calibrated, explainable confidence. It
refuses to assert what the evidence cannot support.
> Detection tells you *that* something happened. Nimbus Vestige tells you *how* — and *what else*.
---
## Why this exists
Modern intrusions do not "break in." They log in. Identity is now the primary attack
vehicle, implicated in the large majority of cloud IR investigations, and most
intrusions span multiple surfaces — identity plus cloud plus SaaS plus endpoint.
The telemetry that records them is fragmented and inconsistent, which already forces
responders to reconstruct the story by hand from incomplete data.
That manual reconstruction is slow, and it fails in a predictable way: the responder
anchors on the first plausible narrative and misses the real one. NV automates the
reconstruction and directly attacks that failure mode by always presenting the ranked
space of plausible paths, not a single story.
Crucially, NV does this with **honesty as the product**. The market's stated enemy is
black-box confidence — a tool that asserts a conclusion without showing its work.
Every number NV produces is traceable to the specific evidence that earned it, and
anything it cannot support is withheld rather than guessed.
---
## What it is — and is not
NV is **not** a detector, a scanner, an auditor, or an attack-runner. Those tools
tell you *that* something happened and measure it. NV works **backward** — abductive
reconstruction of the *mechanism* from patterns, across a fragmented identity estate.
- **Not a detector** — it doesn't fire alerts on activity; it explains how activity fits together.
- **Not a SIEM** — it enters through a narrow wedge (post-incident narrative reconstruction), not as a log platform.
- **Not a rule engine** — when nothing matches a known pattern, it still reconstructs, degrading gracefully instead of going blind.
---
## Why it is valid long-term
1. **Blue team regenerates; red team commoditizes.** Offensive tooling finds a finite,
patchable set of holes and is folding into automated CI/CD safety pipelines.
Reconstructing *how* an intrusion happened never resolves — attackers keep inventing,
so the need is permanent and self-renewing.
2. **The engine is substrate-independent.** The core logic — reconstruct the mechanism
from patterns, with calibrated confidence and a stop rail — is committed to
cloud/identity first, but ports to network, endpoint, and OT later. The target can
change without rewriting the thesis.
3. **Reconstruction enables hardening.** Once you know how they got in — and which other
doors were open — you build the defenses. The untaken-but-plausible paths are a
hardening backlog, often worth more than the reconstruction itself, because most
breaches exploit preventable exposure, not novel tradecraft.
4. **It degrades gracefully on the novel intrusion** — the very incident that matters
most. A signature/rule engine goes blind on a zero-day; NV's abductive core still
produces a most-likely path, honestly flagged as lower-confidence.
5. **Honesty is a moat.** Calibrated, case-based confidence with ranked alternatives is
exactly what the market says it wants and what black-box competitors structurally
cannot offer without redesign.
---
## Architecture
Raw provider logs → normalized event/entity graph → reconstruction engine → JSON → GUI.
```
O365 / Entra audit logs nv_extract_identity_events.py
AWS CloudTrail (IAM/STS/S3) nv_extract_cloudtrail_events.py (ingest + normalize)
▼
normalized identity events (JSONL) ← one event model, any substrate
│ nv/graph.py (typed per-actor timelines)
▼
reconstruction engine
├─ nv/patterns.py known layer: ATT&CK identity pattern library
├─ nv/providers.py provider packs: per-substrate op→ATT&CK vocab (Entra + AWS)
├─ nv/validation.py Phase 3: provenance + integrity gate on every pattern
├─ nv/feeds.py live ATT&CK STIX / TAXII / Sigma clients + scheduler
├─ nv/confidence.py evidence-corroboration scorer (calibratable weights)
├─ nv/calibration.py fit + measure confidence against labeled ground truth
├─ nv/reconstruct.py most-likely chain + ranked competing paths + trust floor
└─ nv/scope.py authorized-scope gate (refuses unauthorized tenants/accounts)
│ run_nv.py
▼
reconstruction.json ──► nv_gui.html (embedding canvas + two-column view + trust slider)
```
The GUI is a **pure view layer** — it renders engine output and lets the analyst move
the trust floor. It contains no reconstruction logic of its own.
---
## The confidence model (the credibility of the whole product)
Every step carries a confidence in [0, 1], built by **evidence corroboration**:
```
confidence = per-technique base rate
+ bonus for each independent corroborating signal
(source IP, device, successful outcome, temporal adjacency, broad-consent flags)
− penalty for missing signals (e.g. no source IP to corroborate origin)
```
- **verified** — confidence ≥ 0.70 *and* two or more independent signals agree.
- **pattern-only** — above the trust floor but weakly or singly supported.
- **withheld** — below the trust floor; shown greyed, never silently guessed or hidden.
Path scores compose their links by geometric mean, so one weak link honestly drags a
path down rather than being averaged away.
Every weight above — the per-technique base rates, the per-signal bonuses, the missing
penalties — lives in a single `PARAMS` table in `nv/confidence.py`. Those are NV's v1
priors, and they are *calibratable*: `nv/calibration.py` can refit them against labeled
ground truth and the engine will load the result, falling back to the v1 priors when no
calibration is supplied (see below).
### The trust floor
The withhold rule, baked into the engine's output contract — not just the UI. Below the
floor, a step is withheld. Dragging the floor is the **honesty-versus-coverage tradeoff
made physical**: up for strict/high-trust, down for permissive/high-coverage. The
default floor is an editorial decision about where NV sits before the analyst touches it.
---
## Calibrating confidence against ground truth
The v1 weights are defensible, but they are priors. `nv/calibration.py` tunes them
against **labeled ground truth** — events where we know which ones were part of the
intrusion and which were benign — and, crucially, measures whether tuning actually
helped, so a calibration is adopted only if it lowers calibration error rather than just
moving numbers around.
Calibrated confidence means one thing: NV's confidence for a step should equal the
probability that the step really was on the attack path. So each labeled event's
confidence is treated as a predicted probability, and the harness fits:
- **per-operation base rates** — the empirical hit rate for that operation, Bayesian-
shrunk toward the v1 prior so small samples don't overfit; and
- **per-signal bonuses and the two penalties** — by bounded coordinate descent that
minimizes Brier score with an L2 pull back toward the v1 values.
The verified/withheld thresholds are policy, not calibration, and are left untouched.
It reports **Brier score, log-loss, ECE, AUC**, a **reliability table**, and a
**trust-floor sweep** (true-step recall vs benign false-positive rate at each floor),
before and after — so the honesty-versus-coverage tradeoff is legible rather than a
single number.
```bash
# calibrate against a real, labeled BadZure / MAAD-AF run
python3 calibrate.py --labeled run_labeled.jsonl --origin live \
--run-label "badzure-2026-07 tenant-x" --out calibration.json
# demonstrate on the bundled documented-shape proxy corpus
python3 calibrate.py --out calibration.json
# run the engine on calibrated weights (opt-in; v1 priors are the default)
python3 run_nv.py events.jsonl out.json --authorize t.onmicrosoft.com \
--calibration calibration.json
```
**Ground truth source.** The honest input is telemetry from a BadZure / MAAD-AF run in a
real tenant, labeled by joining the unified audit log against the attack tool's own
activity record (every event the tool caused is on-chain; everything else is benign — see
`LABELED_SCHEMA` in `nv/calibration.py`). Until you point it at a real run,
`eval/calibration_cases.py` supplies a documented-shape **proxy** corpus, and every
artifact fitted from it is stamped `source="proxy"` in its provenance so a proxy fit can
never be mistaken for a live one. `calibration.proxy.json` is that bundled proxy artifact.
On the bundled proxy corpus the fit is a clear improvement — Brier 0.398 → 0.045, ECE
0.576 → 0.081, AUC 0.81 → 0.91, and the benign false-positive rate at the 0.60 floor
collapses from 0.69 to 0.00 (the v1 priors were badly over-confident on benign activity).
The proxy is deliberately conservative; a live-tenant run is what produces weights you
should actually deploy.
## Trust integrity (Phase 3)
Two first-class controls protect the reconstruction from bad inputs:
- **Authorized-scope gate** (`nv/scope.py`) — NV refuses to run on any estate not on the
authorized list. Run with `--authorize <tenant-domain>` per estate you are permitted
to investigate. A row the gate cannot place in any estate (an actor with no domain
and no tenant or account) is refused too, because it cannot be shown to be in scope.
Pass `--allow-unscoped` only if you are permitted to investigate those rows; they are
still counted in the output.
- **Pattern-source validation** (`nv/validation.py`) — no pattern enters the library
without passing: (1) trusted-source allowlist, (2) a required content checksum,
(3) schema well-formedness, (4) valid ATT&CK-style technique id. Every admitted
pattern keeps an audit record; rejects are logged with a reason. A malformed or
out-of-policy feed entry is stopped upstream before it can manufacture a false
reconstruction. This is the same skepticism the engine applies to evidence, moved to
the patterns themselves.
The checksum is a consistency check: it is computed from a pattern's own content when
the pattern is built, so it catches content that changed afterward (an edited cache or
committed library file). It cannot catch a feed poisoned at the source. For that, pin
the raw sha256 of a feed payload you trust with `update_feeds.py --pin URL=SHA256`, and
any other payload at that URL is refused before validation.
---
## Keeping the knowledge current (staying ahead of attacker evolution)
A reconstruction engine is only as current as its pattern library plus its ability to
handle what the library has never seen. NV addresses staleness on **two independent
fronts**, by design:
### 1. The known layer stays fresh through a validated update pipeline
The library is not hardcoded — `nv/patterns.py` loads every pattern (including the
builtin set) through `nv/validation.py`, so external feeds enter by the same trusted
path. The live network clients that pull these feeds on a schedule are implemented in
`nv/feeds.py` and driven by `update_feeds.py`. Sources, in trust tiers:
- **MITRE ATT&CK (STIX)** — the authoritative technique taxonomy. The connector reads
the ATT&CK STIX index, pulls the latest enterprise release, and refreshes the
authoritative technique name and tactic for every operation NV reasons over —
dropping any whose technique ATT&CK has since revoked or deprecated. A retired
technique is dropped from every source, the builtin set included, so its operations
fall back to the abductive layer (`no known match`) instead of carrying stale
taxonomy. NV keeps ownership of the operation→technique binding and the base-rate
prior; ATT&CK owns the taxonomy.
- **Vetted CTI via TAXII 2.1** — a full discovery → api-root → collection → objects
client. Pulls STIX attack-pattern objects from a vetted CTI collection and refreshes
the techniques NV tracks. Weighted below ATT&CK.
- **Sigma community ruleset** — pulled as a git archive; each cloud/identity rule that
names an O365/Entra operation and carries an `attack.tXXXX` tag becomes an
operation→technique candidate, with a base rate derived from the rule's own severity
`level` and then scaled down by the community source weight. Rules that don't map are
skipped with a reason, never guessed.
On an operation collision the union is ordered by ascending trust before commit, so an
authoritative ATT&CK binding always wins over a community one; lower-tier content is
still admitted and recorded in the audit as corroboration. `update_library(candidates)`
re-ingests the whole set through validation and rebuilds the library atomically, so a
poisoned feed can never leave the library half-updated.
**From update to reconstruction.** A committed run saves the library to
`.nv_feeds/library.json`. The engine uses it when you pass `--feeds-state .nv_feeds`
to `run_nv.py`; without the flag it uses the builtin library. Either way the startup
output says which library ran, and `reconstruction.json` records it under `library`.
The saved patterns go back through validation on load, and if any fail (the file
changed after it was committed) the engine refuses to run rather than use part of it.
**Two allowlists, not one.** Validation already gates on the source tag; the connectors
add a network-layer *host* allowlist, so a connector can only fetch from a trusted host
over TLS — the transport equivalent of the source allowlist, and a guard against a
hijacked or mistyped feed URL. Every fetch records the raw-payload sha256, URL and time
as provenance, and `update_feeds.py` prints each sha256 so you can pin a payload you
trust. A pinned URL that returns anything else fails its pull, and the library keeps
the last good candidates for that feed.
**Why the validation layer matters more as the feeds grow:** the more you automate
updates, the more an auto-update pipeline becomes a trust risk in disguise — a wrong or
poisoned feed injects false patterns and manufactures false reconstructions. NV treats
ingestion as adversarial: allowlist, checksum, schema, and audit trail on every pattern,
every update.
### 2. The abductive layer covers what no feed has yet named
Feeds always lag the newest tradecraft. The abductive core is the hedge: when an
observed operation matches **no** library pattern, NV does not drop it — it reconstructs
the most-likely path from first principles and flags it `no known match` at reduced
confidence. This is validated in the eval suite (`eval/ground_truth_cases.py`), where a
deliberately-unknown managed-identity token-theft step is caught, sequenced correctly,
and honestly down-weighted below the trust floor.
Together these mean NV never fully goes stale: the known layer tracks the published
frontier through a poisoning-resistant pipeline, and the abductive layer keeps the tool
from going blind on the intrusion the frontier hasn't caught up to yet.
### Suggested update cadence
`nv/feeds.py` carries a per-feed cadence and `update_feeds.py` only pulls what is due,
so it can be dropped straight into cron or a systemd timer:
- ATT&CK STIX — 90 days (a few official releases a year).
- CTI/TAXII — 7 days (as the feed publishes), always through validation.
- Sigma — 30 days.
```bash
# run whatever is due, keeping state under ./.nv_feeds (cron-friendly)
python3 update_feeds.py
# force all feeds now but only report what would change
python3 update_feeds.py --force --dry-run
# run fully offline against captured fixtures (no network)
python3 update_feeds.py --offline eval/fixtures --force
# refuse an ATT&CK release unless it matches the sha256 you trust
# (each run prints the sha256 and URL of every payload it fetched)
python3 update_feeds.py --feed attack-stix --force --pin URL=SHA256
# reconstruct with the committed library instead of the builtin one
python3 run_nv.py your_events.jsonl reconstruction.json \
--authorize yourtenant.onmicrosoft.com --feeds-state .nv_feeds
```
After any library change, re-run `eval/run_eval.py` to confirm known attack shapes still
reconstruct, `eval/validation_cases.py` to confirm the gate still rejects bad input, and
`eval/feeds_offline_test.py` to confirm the feed path is intact end to end.
---
## A note on the demo data
**The reconstruction shown in `nv_gui.html` is built from a public research dataset**
— the invictus-ir Office 365 unified audit log corpus — **not from any company's live
tenant.** It exists so the engine can be demonstrated and dogfooded end-to-end without
anyone's cooperation. To use NV for real, an organization wires in its own Entra/M365
audit data, as described below. No proprietary or customer data is included anywhere in
this project.
## Connecting your own data
NV runs wherever you run it; your logs never have to leave your environment. Four steps:
**1 — Export your audit logs.** NV reads the Microsoft 365 unified audit log. Source it
from Microsoft Purview (Audit search → export CSV), the `Search-UnifiedAuditLog`
Exchange Online PowerShell cmdlet, the Microsoft Graph `auditLogs` / `signIns`
endpoints, or a Sentinel/SIEM export of `OfficeActivity` and `SigninLogs`.
**2 — Normalize it** into the event model NV reasons over:
```bash
python3 nv_extract_identity_events.py your_audit_export.csv your_events.jsonl
```
This keeps sign-ins, consent grants, service-principal and role changes, and mailbox
access — including identity operations NV has never named, so they still reach the
abductive layer — and drops the rest. It reads a CSV carrying the standard `AuditData`
column, or JSON/JSONL where each record is an `AuditData` object (the shape the
invictus-ir research corpus ships in).
**3 — Reconstruct, scope-gated.** The engine refuses to run on a tenant you haven't
explicitly authorized:
```bash
python3 run_nv.py your_events.jsonl reconstruction.json \
--authorize yourtenant.onmicrosoft.com
```
Service principals and other actors without an email-style name have no domain, so the
ingest scopes them by the tenant id their record carries, as `m365:<OrganizationId>`.
If the engine lists one of these as unauthorized, add it with a second `--authorize`.
**4 — View.** Open `nv_gui.html` and click **⤒ load reconstruction.json** to point it at
the file the engine just produced — the same screen then shows your incident, your
entities, and your confidence bands. Every dot on the embedding canvas re-runs the
reconstruction for that entity on click.
### Analyzing AWS CloudTrail instead
The engine is substrate-independent — only the operation vocabulary is provider-specific
(`nv/providers.py`). The AWS path is the same three steps against CloudTrail:
```bash
python3 nv_extract_cloudtrail_events.py cloudtrail.json aws_events.jsonl
python3 run_nv.py aws_events.jsonl reconstruction.json --authorize aws:123456789012
```
The ingest keeps IAM/STS/sign-in events (novel IAM operations included, so they reach the
abductive layer) plus S3 object reads, and scope-gates on the AWS **account** rather than a
tenant domain. The same abductive core, confidence model, and trust floor apply unchanged.
The GUI also carries a **demo-data banner** and an in-app "Connecting your own data"
guide, so anyone opening it understands the reconstruction is public sample data until
they wire in their own.
## Project layout
```
run_nv.py reconstruction engine entrypoint (scope-gated)
nv_extract_identity_events.py ingest: M365/Entra unified audit log -> event model
nv_extract_cloudtrail_events.py ingest: AWS CloudTrail -> event model [iteration 3]
update_feeds.py pull + validate threat-intel feeds on a cadence [iteration 1]
calibrate.py fit confidence weights from labeled ground truth [iteration 2]
nv/ the engine package
graph.py patterns.py validation.py confidence.py reconstruct.py scope.py
providers.py per-substrate op->ATT&CK vocabulary packs (Entra + AWS) [iteration 3]
feeds.py ATT&CK STIX / TAXII 2.1 / Sigma clients + scheduler [iteration 1]
calibration.py metrics + parameter fit [iteration 2]
eval/ evals + offline fixtures (no network, no tenant)
nv_gui.html pure view layer (loads engine reconstruction.json)
calibration.proxy.json bundled proxy calibration artifact [iteration 2]
```
Run everything from the project root so the `nv` package is importable.
## Running it
```bash
# 1. normalize raw O365/Entra audit logs into the event model
python3 nv_extract_identity_events.py auditrecords.csv nv_identity_events.jsonl
# 2. reconstruct (scope-gated; add --calibration calibration.json to use fitted weights)
python3 run_nv.py nv_identity_events.jsonl reconstruction.json \
--trust-floor 0.6 --authorize your-tenant.onmicrosoft.com
# 3. view — open nv_gui.html (renders reconstruction.json)
```
Keep the pattern library current and tune confidence:
```bash
python3 update_feeds.py # pull + validate feeds that are due
python3 calibrate.py --labeled run.jsonl --origin live --out calibration.json
```
### Evals
```bash
python3 eval/run_eval.py # attack shapes reconstruct correctly
python3 eval/validation_cases.py # poisoned patterns are rejected
python3 eval/feeds_offline_test.py # feed connectors + scheduler (offline)
python3 eval/providers_aws_test.py # AWS chain reconstructs through the same engine
python3 eval/scope_test.py # scope gate refuses other and unscoped estates
python3 eval/entities_test.py # every canvas entity reconstructs on click
python3 eval/label_join_test.py # tool activity log joins into labeled ground truth
python3 calibrate.py # calibration harness on the bundled proxy corpus
```
Each `eval/` script above exits nonzero when a check fails, and CI runs all of them on
every merge request and on the default branch.
---
## Status
**Working end-to-end on real data** (invictus-ir O365 dataset), across the full plan:
| Phase | Item | Status |
|---|---|---|
| 1 | Normalized event/entity model | done |
| 1 | Ingest + normalization (Entra/M365 beachhead) | done |
| 2 | Known layer (ATT&CK pattern library) | done |
| 2 | Evidence-corroboration confidence model | done |
| 2 | Trust floor in the output contract | done |
| 2 | Abductive core (unknown layer) | done, eval-proven |
| 2 | Ranked competing hypotheses | done |
| 3 | Pattern-source validation layer | done |
| 3 | Authorized-scope control | done |
| 3 | Live feed connectors (ATT&CK STIX / TAXII / Sigma) + scheduler | done |
| 4 | Two-column screen on real engine output | done |
| 4 | Embedding / similarity canvas | done |
| 4 | Canvas-driven per-entity reconstruction (click re-runs the chain) | done |
| 4 | GUI loads engine `reconstruction.json` (point it at your file) | done |
| 4 | Multi-provider substrate (AWS CloudTrail, eval-proven) | done |
| 5 | Aesthetic / theming | done |
| 5 | Confidence-calibration harness + engine integration | done |
### Honest known gaps (next iterations)
- **Live-tenant calibration numbers.** The calibration *harness* is complete and the
interface is proven, but the shipped `calibration.proxy.json` is fitted on documented-
shape proxy data. Deployable weights require running the harness against a real
BadZure / MAAD-AF run — an operational step for the adopter.
- **Feed-driven vocabulary growth.** The connectors refresh the operations NV already
reasons over; letting a feed *introduce* a new operation (and wire it into the
initial/pivot/collection classification) is future work.
- **Second-provider depth.** AWS is in as a proof of substrate-independence (pack +
ingest + eval), but only IAM/STS/S3. Widening the AWS pack, adding provider-specific
feed connectors, and a third substrate (GCP, Okta) are the next steps.
---
## License
Nimbus Vestige is **source-available** under the **PolyForm Noncommercial License
1.0.0** (see [LICENSE](https://gitlab.com/dobybaxter127/nimbus-vestige/-/blob/main/LICENSE)). The whole tool — reconstruction engine, both
provider ingests, GUI, and evaluation harness — is free to inspect, run, and use
for any **noncommercial** purpose: personal projects, research, education,
non-profits, and evaluation. Read every line before you decide.
Nimbus Vestige is an open project, not a commercial product. The license covers
noncommercial use only.
---
### Author
Doby Baxter 2026