DetectTrace

A detection should have fired. It didn't. DetectTrace tells you exactly why.

DetectTrace is an open-source detection engineering tool for testing security detections end to end and localising the first failing stage.

Instead of testing only a SIEM query, DetectTrace treats a detection as a pipeline:

test behaviour
    -> telemetry
    -> ingestion
    -> normalization/schema
    -> rule evaluation
    -> rule execution
    -> alert generation

A DetectSpec declares what should happen. DetectTrace gathers evidence for what actually happened, evaluates the contract, stops causal reasoning at the first proven failure, and marks dependent downstream stages BLOCKED.

The current live backend is Elastic Security. DetectTrace also includes a deterministic file-backed mode for local development and regression testing.

Why DetectTrace exists

Detection failures are often diagnosed manually:

• Did the test actually run?
• Was telemetry generated?
• Did it reach the backend?
• Did a field get renamed or dropped?
• Does the rule still match?
• Is the rule disabled or unhealthy?
• Did the alert appear, and did it belong to this exact test run?

DetectTrace turns those questions into executable checks and evidence.

Example failure:

Test event                 PASS
Backend connection         PASS
Telemetry index            PASS
Telemetry located          PASS
Normalization              FAIL
Rule                       BLOCKED
Elastic rule exists        BLOCKED
Elastic rule enabled       BLOCKED
Rule execution             BLOCKED
Elastic alert              BLOCKED

RESULT
------------------------------------------------------------------------
Required field 'process.command_line' is absent, but the value from
'winlog.event_data.CommandLine' survived at 'process.args'.
Probable schema/mapping drift.

Confidence: HIGH
First failing stage: NORMALIZATION
Failure code: SCHEMA_DRIFT

The important part is not only that the detection failed. DetectTrace explains where the detection path first became invalid and why.

What DetectTrace currently supports

• DetectSpec v1 validation
• deterministic offline traces from JSON/JSONL evidence
• safe declarative predicate DSL; no arbitrary Python or shell execution
• nested field checks
• field-lineage hints for likely schema/mapping drift
• PASS / FAIL / BLOCKED / UNKNOWN / SKIPPED stage semantics
• first-failing-stage diagnostics
• stable failure codes and remediation text
• correlated Elastic Security tests using a unique detecttrace.run_id
• real Elastic rule existence/enabled/execution checks
• correlated Elastic alert verification
• alert latency measurement
• secure Elasticsearch TLS with a configured CA
• detecttrace doctor environment checks
• text, JSON, and JUnit-compatible output
• regression tests and GitHub Actions CI
• reproducible local Elasticsearch + Kibana lab

What DetectTrace is not

DetectTrace is not:

• an attack-simulation platform
• a BAS replacement
• a SIEM rule editor
• a generic vulnerability scanner
• an AI system that guesses whether a detection worked

Attack/test execution can be integrated later. DetectTrace's job is to verify the detection path and diagnose failures from observed evidence.

Requirements

• Python 3.10+
• Docker with Compose v2 only if you want to run the supplied Elastic lab

The CI suite currently tests Python 3.10, 3.11, 3.12, and 3.13.

Five-minute offline quick start

Clone the repository, create a virtual environment, and install DetectTrace.

Windows PowerShell

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e .

Linux/macOS

python -m venv .venv
source .venv/bin/activate
python -m pip install -e .

Check the installation:

detecttrace --version

Create a runnable starter project:

detecttrace init demo

Then:

cd demo
detecttrace validate detectspec.yaml
detecttrace test detectspec.yaml

The generated project is self-contained. It does not require Elasticsearch, Kibana, Docker, or network access.

A healthy run ends with:

Detection contract passed end-to-end.
Confidence: HIGH

Offline healthy and broken examples

The repository includes a PowerShell fixture with both known-good and deliberately broken evidence.

Healthy:

detecttrace test examples/powershell/detectspec.yaml --profile healthy

Broken:

detecttrace test examples/powershell/detectspec.yaml --profile broken

The broken profile deliberately preserves the original command line under process.args instead of the required process.command_line. DetectTrace localises that failure to normalization and blocks rule/alert evaluation.

DetectSpec

DetectSpec is the declarative contract. DetectTrace is the engine that evaluates that contract against evidence.

A DetectSpec can describe:

1. execution evidence
2. source telemetry
3. collection evidence
4. normalization requirements
5. rule semantics
6. alert expectations and timing
7. correlated live test cases

Example:

spec_version: detectspec/v1
id: DET-PS-LIVE-001
title: Live Encoded PowerShell

inputs:
  profiles:
    live: {}

test:
  cases:
    healthy:
      event:
        event:
          code: 1
        process:
          name: powershell.exe
          command_line: powershell.exe -enc AAA

    broken:
      event:
        event:
          code: 1
        winlog:
          event_data:
            CommandLine: powershell.exe -enc AAA
        process:
          name: powershell.exe
          args: powershell.exe -enc AAA

checkpoints:
  normalization:
    require_event:
      all:
        - field: process.name
          op: endswith
          value: powershell.exe
    required_fields:
      - field: process.command_line
        from: winlog.event_data.CommandLine

  rule:
    match:
      all:
        - field: process.name
          op: endswith
          value: powershell.exe
        - field: process.command_line
          op: regex
          value: "(?i)(?:\\s|^)-(?:enc|encodedcommand)\\b"

The JSON Schema is in:

schemas/detectspec-v1.schema.json

The runtime validator and JSON Schema are intentionally strict about unknown DetectSpec structure so spelling mistakes fail early.

Predicate DSL

Leaf predicates use:

field: process.name
op: equals
value: powershell.exe

Supported operators include:

exists
equals
not_equals
contains
startswith
endswith
regex
in
gt
gte
lt
lte

Predicates can be composed with all, any, and not.

Example:

all:
  - field: process.name
    op: endswith
    value: powershell.exe
  - any:
      - field: process.command_line
        op: contains
        value: "-enc"
      - field: process.command_line
        op: contains
        value: "-EncodedCommand"

First-failing-stage semantics

DetectTrace follows a simple rule:

Evidence before inference. First failing stage wins.

If normalization fails, DetectTrace does not pretend it knows whether a downstream rule or alert would have succeeded. Those stages are reported as BLOCKED.

If evidence is unavailable rather than disproven, DetectTrace reports UNKNOWN instead of guessing.

The stable machine-readable result schema is:

detecttrace.result/v1

Important fields include:

healthy
first_failed_stage
failure_code
confidence
root_cause
remediation
run_id
stages

Current failure codes include categories such as:

INGESTION_FAILURE
TELEMETRY_MISSING
SCHEMA_DRIFT
REQUIRED_FIELD_MISSING
RULE_NOT_FOUND
RULE_DISABLED
RULE_LOGIC_MISMATCH
RULE_EXECUTION_ERROR
ALERT_TIMEOUT
UNKNOWN

Local Elastic Security lab

A reproducible secured lab is included at:

lab/elastic

It provides: