log-horizon

Disclaimer

[!CAUTION] Disclaimer

This tool is developed and maintained with the help of AI. Please exercise caution when using this solution and always understand what are you running before you run it in production. The developer assumes no liability for any vulnerabilities or issues.

By downloading, installing, or using this tool, you acknowledge that you have read, understood, and agree to these terms.

Prerequisites

Endpoints follow the Azure environment of the current Connect-AzAccount session, so Azure Government and Azure China workspaces work without extra parameters. The public cloud values are used when no environment is available.

If you're not already logged into Azure, the module will fire up Connect-AzAccount for you. If you are, it'll just carry on.

Permissions: Log Analytics Reader and Microsoft Sentinel Reader on the workspace cover the analysis. Transform discovery also needs Microsoft.Insights/dataCollectionRules/read (Monitoring Reader) on the subscription or resource group; without it the run continues and prints a warning naming the missing permission. -IncludeDefenderXDR uses Microsoft Graph with CustomDetection.Read.All, which for a signed-in user means the optional Microsoft.Graph.Authentication module.

Getting started

Pretty straight forward:

Usage

The basics

Start by connecting to Azure and making sure you select the right account and subscription:

Then we can invoke the tool:

Output should look something like this:

Also has a menu to dig deeper into other outputs:

Keyword gaps + Defender XDR

Want to know if you're missing tables related to specific vendors? Throw in some keywords. Add -IncludeDefenderXDR if you want the XDR analysis too.

Detection Analyzer

Enable rule quality/noise analysis based on incidents and automation rules:

Export a report

Manage table retention and type

You can now update table retention and table type directly from the interactive TUI:

• Open Invoke-LogHorizon normally, then choose Manage table retention and type from the main menu for bulk retention or type updates.
• Open Log Tuning / Transforms > Evaluate specific table and choose Manage retention/type for this table for a single-table change.

For scripting or automation, use the dedicated public command:

Non-interactive / CI mode

Skip the interactive TUI and export straight to a file, useful for pipelines or scheduled runs:

If you omit -Output, the analysis object is returned to the pipeline so you can pipe it into your own logic.

Collection cache

The data collection phase (usage, rules, incidents, tables, DCRs) is cached by default so re-running against the same workspace, for example to export a second format or to reopen the TUI, takes seconds instead of minutes. The cache lives under $env:LOCALAPPDATA\LogHorizon\cache (override with -CachePath), one file per combination of subscription, resource group, workspace, -DaysBack, -DetectionLookbackDays, -IncludeDefenderXDR, -IncludeDetectionAnalyzer, the three price parameters and the module version. Entries older than -CacheMaxAgeMinutes (default 60) are ignored, and every save deletes expired entries so the folder does not accumulate files from other parameter sets or older versions. Tokens are never written to the cache; authentication runs on every invocation so the retention wizard always has live credentials.

The cache file is plaintext Clixml. With -IncludeDetectionAnalyzer it contains incident titles, numbers, status and classification from your workspace; incident owners, rule authors and assigned-owner identities are not collected. Use -NoCache on shared machines or point -CachePath at a location with the access control you need.

Log tuning and split KQL

The interactive TUI includes a Log Tuning / Transforms menu with live tuning suggestions (field usage from your deployed rules and hunting queries), knowledge-base split KQL, and a per-table evaluator. Every KQL block is condition-only and pastes straight into the Sentinel split rule editor, with source attribution (knowledge base, rule analysis, or combined).

Custom pricing

Default prices are West Europe Simplified PAYG in USD: 5.59 $/GB Analytics, 1.15 $/GB Basic, 0.20 $/GB Data Lake (0.07 ingestion + 0.13 processing). Each table is priced by the plan observed in the Usage table, and free status comes from Usage.IsBillable. Volumes use billing GB (1000 MB). If your commitment tier is different:

All parameters

Under the hood

So there's four phases.

1. Data collection

The module connects to Azure and pulls data from the Log Analytics, Security Insights, Azure Monitor and Microsoft Graph APIs. Every call goes through a retry wrapper (429, 5xx and transport errors) and, unless -NoCache is set, the collected data is cached locally for the next run.

2. Classification

Every table gets classified through two passes:

First, a direct lookup against the 481-entry knowledge base in Data/log-classifications.json. Each entry carries the connector name, primary/secondary classification, security category, MITRE data source mappings, a recommended pricing tier and retention, and optional lifecycle flags (deprecated/legacy with replacement tables, XDR streamability, platform).

If there's no match, heuristic rules kick in, in this order:

• Name contains a security token such as Alert, Incident, Threat, Signin, Logon, Audit, Risk, Detection (matched at PascalCase word starts) -> primary
• Name looks like infra telemetry: Flow, Metric, Diagnostic, Perf, Heartbeat, Health, Inventory, Usage -> secondary
• Has active analytics rules pointing at it -> primary
• Built-in table with a Microsoft prefix (AAD, Microsoft, Azure, Defender, Purview, Entra, Sentinel, Office, Intune, Windows ...) -> primary, flagged for review and addition to the database
• Generic *Log/*Logs name -> secondary
• High volume (>10 GB/mo) with nothing detecting on it -> secondary
• None of the above -> unknown

3. Cost-value scoring

Each table gets scored on a few dimensions:

• Cost tier: Free / Low (<1 GB) / Medium (1-10 GB) / High (10-50 GB) / Very High (>50 GB)
• Detection tier: None / Low (1-2 rules) / Medium (3-9 rules) / High (10+ rules)
• Assessment: High Value / Good Value / Missing Coverage / Optimize / Low Value / Underutilized / Free Tier / Platform
• Coverage %: Percentage of tables with at least one analytics rule or hunting query referencing them, calculated as tablesWithRules / totalTables * 100. Per-table coverage sums analytics rules + hunting queries found by parsing KQL for table names.
• Implicit coverage: Rule kinds that carry no KQL still consume tables. Data/implicit-consumers.json maps them (Threat Intelligence matching -> ThreatIntelIndicators/ThreatIntelObjects, Fusion -> SecurityAlert/Anomalies, UEBA -> BehaviorAnalytics/UserPeerAnalytics/IdentityInfo, Microsoft incident creation -> SecurityAlert). Enabled rules of those kinds count toward effective coverage, and each table reports a CoverageSource of kql, xdr, implicit, platform or none. Platform tables Sentinel writes for itself (SecurityIncident, SentinelHealth, Watchlist, Usage ...) are never flagged as missing coverage and get the Platform assessment.
• Only enabled analytics rules and enabled Defender custom detections count toward coverage.

Then the module generates recommendations (13 types):

Recommendations are sorted once, High > Medium > Low and then by estimated savings, and every output (JSON, Markdown, HTML, TUI) keeps that order.

4. Detection Analyzer (noisiness scoring)

When you pass -IncludeDetectionAnalyzer, the module fetches recent incidents and automation rules, then scores every enabled analytics rule for potential noisiness.

Per-rule metrics (computed from incident data):

Noisiness score formula:

Each metric is converted to a percentile rank across all rules that have at least one incident. The composite score is a weighted blend:

• Volume percentile (35%): how many incidents a rule generates relative to other rules.
• AutoClose percentile (40%): how often incidents are auto-closed by automation rules (highest weight because automated closure is the strongest signal of low-value alerts).
• FalsePositive percentile (25%): how often analysts classify the outcome as false positive.

Score thresholds:

Incidents are bucketed by analytic rule id (falling back to rule name, then title), so two rules sharing a display name are scored separately.

Rules with a score ≥ 70 and at least 5 incidents are automatically surfaced as High-priority recommendations in the Recommendations view.

5. Interactive dashboard

The main menu offers these views:

• Dashboard: overview stats, top 10 costliest tables (deprecated and legacy sources carry a badge), coverage bar, retention compliance summary, correlation exclusion callout
• View Recommendations: prioritised actions with estimated monthly savings, expandable to show the full list when there are more than 10
• View Detection Assessment: cost-value matrix summary, per-table rule and hunting query coverage, primary/secondary drill-down, correlation-excluded rule listing
• View Detection Analyzer: percentile-based noisy rule ranking with closure quality indicators (when you used -IncludeDetectionAnalyzer), searchable rule browser
• View SOC Optimization: Microsoft's own improvement suggestions with drill-down
• View Retention Assessment: tables below recommended minimums with current vs recommended retention, plan type, and shortfall, plus XDR advisory rows
• View Data Transforms: DCR transform inventory with transform type classification and full KQL per table
• Log Tuning / Transforms: live tuning suggestions, knowledge-base split KQL, and a per-table evaluator with a single-table retention/type change
• View All Tables: the full list with classification, plans, cost, rules, retention (colour-coded), and assessment; pick a table for a detail panel (coverage sources, plan support, retention, status, its recommendations)
• Manage table retention and type: bulk retention and plan wizard with preview and apply
• Dictionary: every term the tool uses (classification, cost and detection tiers, assessments, coverage sources, the 13 recommendation types, Detection Analyzer metrics and score labels, table plans, lifecycle status, XDR states, transform types) with the same definitions as this README, served from Data/dictionary.json
• Export Report: pick a format, then a path (directory for a timestamped file, or a file name; Enter keeps the current directory) and write JSON, Markdown or HTML right from the menu
• XDR Analysis appears on the dashboard when you used -IncludeDefenderXDR

The classification database

Sitting at Data/log-classifications.json. 481 entries, 243 connectors, 22 categories.

What's in each entry

Tables with a status show a badge in the TUI and the reports, and any that still ingest raise a DeprecatedSource recommendation naming the replacement. Tables with xdrStreamable: false are never treated as XDR streaming candidates by the XDR Checker.

Primary vs secondary security data

Primary (278 entries): the tables you're actually building detections on. Sign-in logs, security alerts, threat intel, audit trails, vulnerability findings, firewall hits, EDR telemetry.

Secondary (203 entries): supporting stuff. Perf metrics, infrastructure diagnostics, network flow volumes, inventory snapshots, config baselines, health checks.

Categories at a glance

Table plan support

Data/basic-plan-tables.json and Data/auxiliary-plan-tables.json list the built-in tables that the Azure Monitor table feature matrix marks as supporting the Basic and Auxiliary (Data Lake) plans. The retention wizard uses the Basic list to validate plan switches, and the DataLake recommendation only proposes the lake tier for tables that support it, falling back to a Basic plan suggestion where that is the lowest supported tier. DCR-based custom tables support both plans; Classic custom tables support neither. Both files are regenerated from that matrix for each release.

Custom classifications

You can provide your own classification file to add entries for tables not in the built-in database, or override existing entries when the defaults don't match your environment. Custom entries take precedence over built-in ones when the same tableName appears in both.

The custom file uses the same schema as Data/log-classifications.json, an array of objects:

See Data/custom-classifications-example.json for a ready-to-use template.

How the classifications were built

The primary/secondary grading was done partially by the author and then by feeding Microsoft's data connector and table definitions into AI with a human grading baseline, using Microsoft best practices and industry standards as the classification criteria.If something looks off for your environment, trust your own context over the tool - AI can make mistakes, and context is king.

The classification criteria were drawn from the following sources:

ACSC (Australian Signals Directorate, Australian Cyber Security Centre)

• ACSC: Best practices for event logging and threat detection (Aug 2024)
• ACSC: Priority logs for SIEM ingestion - Practitioner guidance (May 2025)

CISA (Cybersecurity and Infrastructure Security Agency)

• CISA: Guidance for Implementing M-21-31: Improving the Federal Government's Investigative and Remediation Capabilities
• CISA: Microsoft Expanded Cloud Logs Implementation Playbook (2025)

Microsoft

• Microsoft Sentinel data connectors reference
• Microsoft Sentinel tables & connectors reference
• Azure-Sentinel GitHub repo (community analytics rules, connector definitions, solution templates)
• Microsoft Sentinel billing
• Microsoft Sentinel data tier management

MITRE

• MITRE ATT&CK Data Sources

NIST (National Institute of Standards and Technology)

• NIST SP 800-92: Guide to Computer Security Log Management

NSA (National Security Agency)

• NSA Cyber Event Forwarding Guidance

NCSC-UK (National Cyber Security Centre - United Kingdom)

• NCSC-UK's "What exactly should we be logging?"

Google Cloud

• Google Cloud Audit Logs overview
• Google Cloud Audit Logs best practices

Other sources were also used, along with the authors "expertise" if you can categorize it as such.

Project layout

How the knowledge bases are generated

The split KQL knowledge bases, high-value-fields.json and field-frequency-stats.json, are built offline from the public Azure/Azure-Sentinel repository (Solutions/, Detections/, Hunting Queries/, roughly 3,800 YAML rule files). Each query is run through the same Get-TablesFromKql and Get-FieldsFromKql the module uses at runtime to build per-table field frequency counts, from which three tiers of fallback fields are derived:

• Universal fields: fields appearing in more than 50% of all tables (for example TimeGenerated)
• Category defaults: fields appearing in more than 40% of tables within a classification category
• Per-table stats: raw field frequency counts for tables with at least 3 referencing rules

Mined fields are merged into the curated high-value-fields.json (curated entries are kept, newly discovered tables with at least 3 rules and 3 meaningful fields are added).

At runtime, Get-SplitKql uses a fallback hierarchy: curated KB entry -> live rule/hunting field analysis -> community per-table stats -> category defaults -> universal fields. Frequency of fields is not a perfect method, but it's useful to know.

Tests

437 Pester v5 tests, no Azure connectivity required. Run them from a plain PowerShell session rather than the VS Code integrated terminal:

License

GPL-3.0. See LICENSE.

Version history

Known issues

PwshSpectreConsole UTF-8 encoding warning

To enable UTF-8 output in your terminal, add the following line at the top of your PowerShell $PROFILE file and restart the terminal:

The module sets this automatically on import, but depending on your session the warning may still appear. It's cosmetic and doesn't affect functionality.

Contributing

If you have issues or want to contribute, create a PR.

Like the tool?