tracebyte8/systraceLinux system-call monitor using ptrace to trace file, process, network, and memory activity, with namespace isolation and machine learning classification for behavioral security analysis.
ptrace-based Linux syscall monitor and sandbox for analyzing ELF binaries.
SysTrace runs a target inside an isolated Linux namespace environment, traces its system calls and child processes, applies rule-based detection and weighted risk scoring, then uses a Random Forest classifier as a secondary signal.
ptrace syscall tracingfork, vfork, clone, and execvepivot_root sandboxSIGKILL enforcementClone the repository:
git clone https://github.com/tracebyte8/SysTrace.git
cd SysTrace
Make the configuration script executable:
chmod +x config.sh
Run SysTrace against a compiled ELF binary:
./config.sh ./tests/bin/mal_fileopen
config.sh automatically:
gcc, make, and python3.venvNo manual Python package installation is required.
The target must be an already compiled ELF binary. A
.csource file cannot be passed directly.
Target ELF
│
▼
Namespace Sandbox
│
├── CLONE_NEWUSER
├── CLONE_NEWPID
├── CLONE_NEWNS
└── pivot_root
│
▼
ptrace Tracer
│
├── Syscall tracing
├── fork/vfork/clone following
└── exec following
│
▼
System Call Monitors
│
├── File
├── Process
├── Memory
└── Network
│
▼
Statistics
│
▼
Rule Engine
│
├── Alerts
└── SIGKILL enforcement
│
▼
Risk Scoring
│
├── features.json
│
└── Random Forest
│
▼
security_report.html
| Category | Tracked syscalls |
|---|---|
| File | open, openat, read, close |
| Process | execve, fork, clone, wait4, ptrace |
| Memory | mmap, mprotect |
| Network | socket, connect, sendto, recvfrom, bind, listen, accept |
Other syscalls are also written to syscall.txt when their names are available in the syscall table.
| Trigger | Condition | Action |
|---|---|---|
| Sensitive file access | /etc/passwd or /etc/shadow | Alert + SIGKILL |
| Excessive file opens | open > 100 | Alert + SIGKILL |
| Excessive reads | read > 29 | Alert + SIGKILL |
| Excessive forking | fork/clone > 8 | Alert + SIGKILL |
| Excessive re-execution | execve > 8 | Alert + SIGKILL |
| Network connection | Any connect() | Alert + SIGKILL |
| Network send | Tracked send syscall | Alert + SIGKILL |
| Memory protection change | mprotect > 5 | Alert + SIGKILL |
| Cross-process tracing | Any ptrace() | Alert + SIGKILL |
| High risk | Score ≥ 70 | Alert + SIGKILL |
| Moderate risk | 40 ≤ score < 70 | Alert |
killit records SIGKILL actions performed by the rule engine.
SysTrace calculates a weighted behavioral score:
score =
(sum of weighted syscall counts)
/ (total syscalls × 6.0)
× 100
The score is clamped to 0–100.
If the rule engine kills a process, the risk score is forced to at least 90.
| Behavior | Weight |
|---|---|
ptrace | 6.0 |
connect | 5.0 |
| network | 4.0 |
execve | 3.0 |
mprotect | 3.0 |
fork | 2.0 |
| process | 1.5 |
open | 0.5 |
mmap | 0.5 |
| file | 0.3 |
read | 0.2 |
close | 0.1 |
SysTrace uses a Random Forest classifier as a secondary behavioral signal.
The model is located at:
ml/syscall_model.pkl
The monitored syscall statistics are exported to:
features.json
ml/predict.py reads the latest feature record and produces:
prediction.txt
Example:
Program: ./tests/bin/mal_fileopen
Prediction: MALICIOUS
Confidence: 91.42%
The ML result should be treated as a secondary signal, not a definitive verdict.
The repository does not ship the original training dataset or published model accuracy.
| File | Description |
|---|---|
security_report.html | Final HTML security report |
alerts.json | JSON security alerts |
features.json | ML feature records |
prediction.txt | Latest ML prediction |
log.txt | Human-readable alerts |
syscall.txt | Raw syscall trace |
features.json, alerts.json, log.txt, and syscall.txt accumulate records across runs.
security_report.html and prediction.txt are overwritten on each run.
The HTML report contains:
SysTrace/
├── src/
│ ├── tracer.c
│ ├── namespace.c
│ ├── set_root.c
│ ├── file_monitor.c
│ ├── process_monitor.c
│ ├── memory_monitor.c
│ ├── network_monitor.c
│ ├── rules.c
│ ├── score.c
│ ├── stat.c
│ └── dataset.c
│
├── include/
├── dashboard/
│ ├── index.c
│ └── style.css
│
├── ml/
│ ├── train.py
│ ├── predict.py
│ └── syscall_model.pkl
│
├── tests/
│ └── bin/
│
├── image/
├── Makefile
├── config.sh
└── README.md
If you do not want to use config.sh:
make
Clean the project:
make clean
Rebuild:
make re
Manual Python setup:
python3 -m venv .venv
source .venv/bin/activate
pip install numpy scikit-learn
| Program | Behavior |
|---|---|
benign_idle.c | Sleeps and exits |
benign_fileread.c | Creates, reads, and removes a temporary file |
mal_fileopen.c | Opens/reads/closes multiple files |
mal_forkbomb.c | Creates a capped number of children |
mal_connect.c | Attempts multiple network connections |
mal_mmap_mprotect.c | Repeated mmap/mprotect operations |
Example:
./config.sh ./tests/bin/mal_forkbomb
Then open:
xdg-open security_report.html
SysTrace is a research/learning dynamic-analysis tool, not a hardened security boundary or production EDR.
Important limitations:
SIGKILL after detection; it is not kernel-level syscall blocking.pivot_root.Run SysTrace only against binaries you own or are authorized to analyze.
For untrusted binaries, use a dedicated VM or other appropriately isolated environment.
SysTrace 2.0