SysTrace 2.0

ptrace-based Linux syscall monitor and sandbox for analyzing ELF binaries.

SysTrace runs a target inside an isolated Linux namespace environment, traces its system calls and child processes, applies rule-based detection and weighted risk scoring, then uses a Random Forest classifier as a secondary signal.

Features

• ptrace syscall tracing
• Follows fork, vfork, clone, and execve
• Linux user, PID, and mount namespaces
• Minimal pivot_root sandbox
• File, process, memory, and network monitoring
• Per-process and aggregate syscall statistics
• Rule-based detection with SIGKILL enforcement
• Weighted risk score from 0–100
• Random Forest ML classification
• HTML security report
• JSON and text logs
• Raw syscall trace

Quick Start

Clone the repository:

git clone https://github.com/tracebyte8/SysTrace.git
cd SysTrace

Make the configuration script executable:

chmod +x config.sh

Run SysTrace against a compiled ELF binary:

./config.sh ./tests/bin/mal_fileopen

config.sh automatically:

1. Checks gcc, make, and python3
2. Creates .venv
3. Installs the required Python dependencies
4. Builds SysTrace
5. Runs the target through the monitor
6. Generates the security report

No manual Python package installation is required.

The target must be an already compiled ELF binary. A .c source file cannot be passed directly.

Architecture

Target ELF
    │
    ▼
Namespace Sandbox
    │
    ├── CLONE_NEWUSER
    ├── CLONE_NEWPID
    ├── CLONE_NEWNS
    └── pivot_root
    │
    ▼
ptrace Tracer
    │
    ├── Syscall tracing
    ├── fork/vfork/clone following
    └── exec following
    │
    ▼
System Call Monitors
    │
    ├── File
    ├── Process
    ├── Memory
    └── Network
    │
    ▼
Statistics
    │
    ▼
Rule Engine
    │
    ├── Alerts
    └── SIGKILL enforcement
    │
    ▼
Risk Scoring
    │
    ├── features.json
    │
    └── Random Forest
             │
             ▼
      security_report.html

What It Monitors

Other syscalls are also written to syscall.txt when their names are available in the syscall table.

Detection Rules

killit records SIGKILL actions performed by the rule engine.

Risk Scoring

SysTrace calculates a weighted behavioral score:

score =
    (sum of weighted syscall counts)
    / (total syscalls × 6.0)
    × 100

The score is clamped to 0–100.

If the rule engine kills a process, the risk score is forced to at least 90.

Weights

Machine Learning

SysTrace uses a Random Forest classifier as a secondary behavioral signal.

The model is located at:

ml/syscall_model.pkl

The monitored syscall statistics are exported to:

features.json

ml/predict.py reads the latest feature record and produces:

prediction.txt

Example:

Program: ./tests/bin/mal_fileopen

Prediction: MALICIOUS
Confidence: 91.42%

The ML result should be treated as a secondary signal, not a definitive verdict.

The repository does not ship the original training dataset or published model accuracy.

Reports and Logs

features.json, alerts.json, log.txt, and syscall.txt accumulate records across runs.

security_report.html and prediction.txt are overwritten on each run.

Dashboard

The HTML report contains:

• Rule engine score
• ML prediction
• ML confidence
• Final danger percentage
• Security events
• Behavioral statistics

Project Structure

SysTrace/
├── src/
│   ├── tracer.c
│   ├── namespace.c
│   ├── set_root.c
│   ├── file_monitor.c
│   ├── process_monitor.c
│   ├── memory_monitor.c
│   ├── network_monitor.c
│   ├── rules.c
│   ├── score.c
│   ├── stat.c
│   └── dataset.c
│
├── include/
├── dashboard/
│   ├── index.c
│   └── style.css
│
├── ml/
│   ├── train.py
│   ├── predict.py
│   └── syscall_model.pkl
│
├── tests/
│   └── bin/
│
├── image/
├── Makefile
├── config.sh
└── README.md

Build Manually

If you do not want to use config.sh:

make

Clean the project:

make clean

Rebuild:

make re

Manual Python setup:

python3 -m venv .venv
source .venv/bin/activate
pip install numpy scikit-learn

Test Programs

Example:

./config.sh ./tests/bin/mal_forkbomb

Then open:

xdg-open security_report.html

Security and Limitations

SysTrace is a research/learning dynamic-analysis tool, not a hardened security boundary or production EDR.

Important limitations:

• Only a single target is analyzed per run.
• Enforcement occurs through SIGKILL after detection; it is not kernel-level syscall blocking.
• The sandbox uses Linux user/PID/mount namespaces and pivot_root.
• Unprivileged user namespaces must be enabled.
• The minimal root filesystem contains only the target and selected runtime libraries.
• Dynamically linked binaries with additional dependencies may fail inside the sandbox.
• The sandbox runtime currently assumes x86-64 library paths.
• File-descriptor tracking is global rather than PID-scoped.
• The ML model has no published accuracy guarantee.
• The rule engine and ML classifier can produce false positives and false negatives.
• The sandbox should not be treated as a replacement for a VM or hardened container.

Run SysTrace only against binaries you own or are authorized to analyze.

For untrusted binaries, use a dedicated VM or other appropriately isolated environment.

Version

SysTrace 2.0