QuasarNix: Adversarially Robust Living-off-the-Land Reverse-Shell Detection

Dmitrijs Trizna · Luca Demetrio · Battista Biggio · Fabio Roli

Overview

Linux living-off-the-land (LOTL) reverse shells abuse legitimate binaries (bash, python, nc, …) to establish covert outbound connections, making signature-based detection unreliable against novel variants and evasion attempts. QuasarNix addresses two open gaps in the field:

1. No public ML-based SIEM detectors — we release the first production-ready, openly licensed ML models for LOTL reverse-shell detection.
2. Adversarial fragility — we evaluate models under evasion and poisoning attacks, and provide adversarially-trained checkpoints that survive all tested attacks.

The framework synthesises a 1M-command training corpus from 34 reverse-shell templates and evaluates 14 model architectures at an operating point of FPR = 10⁻⁶ — reflecting real SIEM alert fatigue constraints.

Key Results

Performance on the held-out test set across baseline heuristics and QuasarNix architectures. TPR is reported as mean ± std across ten independent training runs at FPR = 10⁻⁶. Bold marks the best result; asterisk (*) highlights models discussed in the paper.

Baselines

QuasarNix — Tabular Models (One-Hot Encoding)

QuasarNix — Sequential Models (Token Embeddings)

Takeaway: GBDT achieves 60% TPR at FPR = 10⁻⁶ — 18× higher than signatures (3.37%) — while training in 14 seconds on commodity hardware.

Adversarial Robustness

Shell Escape Perturbations

The table below catalogs Linux shell escape techniques from the attacker's toolkit. The third column indicates whether the technique survives auditd kernel telemetry normalization — only the four bold entries produce a distinct EXECVE record and constitute the true attack surface.

Only 4 of 15 techniques survive kernel-level normalization and are used as the adversarial attack space.

Evasion & Adversarial Training

Three attack families are evaluated — benign content injection, shell escape perturbations, and a hybrid of both:

• Benign content injection devastates all neural models; GBDT maintains ≥93% accuracy due to its feature-importance weighting.
• Shell escape perturbations reduce GBDT and CNN to 0% accuracy at maximum perturbation budget without defenses.
• Adversarial training renders all three attack types ineffective across every evaluated architecture.

Poisoning Robustness

Beyond inference-time evasion, we evaluate training-time attacks:

Label-flipping pollution (0–20% of training labels flipped): models degrade gracefully; GBDT shows inherent resistance through ensemble voting, remaining functional at high pollution ratios.

Backdoor attack (0.01–1% poison ratio, 2–10 token triggers): short triggers (2–4 tokens) fail to install reliable backdoors due to their prevalence in benign traffic. Optimal backdoor installation requires 6–10 token triggers at ≥0.03% poison ratio.

Takeaway: Poisoning attacks require substantial, statistically detectable data injection to succeed. GBDT's ensemble mechanism provides inherent robustness at no adversarial-training cost.

Industry Validation

After this work was released, Google published a conceptually aligned production system at CAMLIS 2025 (arXiv:2512.08802): a two-stage YARA + ML pipeline deployed across tens of thousands of systems, processing up to 250 billion events per day. That system independently validates the hybrid ML-for-SIEM-detection paradigm and the active-learning feedback loop proposed here, demonstrating its viability at industrial scale.

Dataset & Pre-trained Models

from datasets import load_dataset
ds = load_dataset("dtrizna/QuasarNix")

Repository Layout

src/
  augmentation.py       rule-based and generative data synthesis
  evasion.py            white-box / black-box adversarial attacks
  models.py             model definitions (CNN, LSTM, Transformer, XGBoost, …)
  preprocessors.py      tokenisers and feature builders
  scoring.py            evaluation metrics at fixed FPR

experiments/
  ablation_*.py         ablation studies (tokenizer, vocab size, embedding)
  adversarial_*.py      attack and adversarial-training pipelines
  train_release_models.py  end-to-end training script
  logs_*/               TensorBoard runs, CSV metrics, model checkpoints

data/
  signatures/           Sigma rules generated by evolutionary search
  nix_shell/            raw benign command corpus
  powershell/           cross-platform command samples

img/                    publication-ready plots

Environment Setup

uv venv                    # creates .venv (add --python 3.11 for a specific interpreter)
source .venv/bin/activate
uv sync                    # installs dependencies from pyproject.toml

Citation

@article{trizna2025quasarnix,
  author    = {Trizna, Dmitrijs and Demetrio, Luca and Biggio, Battista and Roli, Fabio},
  title     = {Robust Large-Scale Detection of Living-Off-the-Land Reverse Shells via Data Synthesis},
  journal   = {ACM Transactions on Privacy and Security},
  year      = {2025},
  doi       = {10.1145/3807450},
  url       = {https://dl.acm.org/doi/10.1145/3807450}
}