SysTrace

A Linux system-call monitoring and behavioral security analysis tool combining ptrace-based tracing, lightweight namespace isolation, and machine learning classification.

SysTrace observes the runtime behavior of a target program at the system-call level, extracts behavioral features, and classifies the observed activity as benign or malicious using a trained Random Forest model. It is built primarily in C for low-level tracing and isolation, with a Python/scikit-learn component for classification and reporting.

Overview

SysTrace traces a target process using ptrace, capturing system calls related to file access, process creation, network activity, and memory operations. The captured behavior is aggregated into syscall statistics and a feature vector, which is passed to a trained Random Forest classifier to produce a benign/malicious prediction. Results are compiled into a structured HTML security report.

The project also includes a lightweight sandboxing layer built on Linux namespaces, used to isolate the target process during tracing.

Architecture

Core Features

System-Call Tracing

• Built on Linux ptrace to attach to and inspect target processes, including syscall arguments and registers.
• Tracks a broad set of syscalls across four behavioral categories:
  • File operations — open/openat, read, close, chmod
  • Process activity — execve, fork, clone, kill
  • Network activity — socket, connect
  • Memory activity — mmap, mprotect

Rule-Based Detection

Heuristic rules flag suspicious behavioral patterns, including:

• Access to sensitive files
• Excessive file operations
• Excessive process creation
• Suspicious memory behavior (e.g. unusual mmap/mprotect usage)
• Suspicious network activity

Lightweight Sandbox / Isolation Layer

An isolation layer built directly on Linux namespace primitives:

• clone() with CLONE_NEWUSER, CLONE_NEWPID, CLONE_NEWNS
• Mount namespace configuration via MS_PRIVATE
• Root filesystem switching with pivot_root and umount2
• tmpfs-backed minimal root filesystem, constructed at runtime under /tmp/systrace-root

This provides basic process, PID, and mount isolation for the traced target — it is not a container runtime replacement.

Machine Learning Classification

• Feature vectors are derived from syscall statistics, with fields including: file, process, network, fork, connect, execve, read, open, close, memory, chmod, kill, label
• A RandomForestClassifier (scikit-learn) is trained on these vectors to distinguish benign from malicious behavior.
• The trained model is serialized to ml/syscall_model.pkl.
• The DATASET

Reporting

• Structured output as features.json (raw feature data) and report.html (human-readable report).
• The HTML report summarizes monitored activity, triggered security rules/events, syscall statistics, and the final ML classification.

Example Feature Vector

[file, process, network, fork, connect, execve, read, open, close, memory, chmod, kill]
[15,   2,       0,       0,    0,       2,      1,    10,   4,     14,     6,     0]

Technologies

Project Structure

linux-syscall-monitor/
├── include/
├── src/
│   ├── tracer.c            # Core ptrace-based syscall tracer
│   ├── syscall.c           # Syscall interception/handling
│   ├── namespace.c         # Namespace/clone setup
│   ├── set_root.c          # pivot_root / minimal rootfs setup
│   ├── file_monitor.c      # File operation monitoring
│   ├── process_monitor.c   # Process creation/execution monitoring
│   ├── network_monitor.c   # Network activity monitoring
│   ├── memory_monitor.c    # Memory-related activity monitoring
│   ├── fd_tables.c         # File descriptor tracking
│   ├── rules.c             # Rule-based detection logic
│   ├── alert.c             # Security alert generation
│   ├── stat.c               # Syscall statistics aggregation
│   └── dataset.c           # Feature vector / dataset generation
├── ml/
│   ├── train.py             # Random Forest training script
│   ├── predict.py           # Classification / inference script
│   └── syscall_model.pkl    # Trained model
├── dashboard/                # Report/dashboard assets
├── tests/
├── Makefile
├── dataset.csv
└── README.md

Limitations

• The sandbox is an educational, lightweight isolation layer — it is not a production-grade container runtime and is not a hardened malware analysis sandbox.
• Namespace isolation covers user, PID, and mount namespaces only; it does not implement network namespace isolation, seccomp filtering, or cgroup resource limits.
• ptrace-based tracing introduces observable overhead and is detectable/evadable by sufficiently sophisticated malware.
• The ML classifier is trained on a limited feature set (aggregate syscall counts) and dataset; classification accuracy depends on the quality and diversity of training data.
• Not intended for tracing untrusted or genuinely malicious binaries outside of a properly isolated research environment.

Security Disclaimer

SysTrace is a research and learning project exploring Linux internals, system-call tracing, namespace-based isolation, and behavioral malware detection using machine learning. It is not a certified or production-ready security tool and should not be relied upon as the sole defense against malicious software. Use in isolated, controlled environments only.

Purpose

This project was built to explore and demonstrate:

• Linux internals and system-call tracing with ptrace
• Process, file, network, and memory monitoring
• Filesystem isolation using Linux namespaces (clone, pivot_root, mount namespaces)
• Sandbox architecture design
• Behavioral security analysis
• Applying machine learning (Random Forest) to cybersecurity classification problems