CVE-2021-25741 — TOCTOU Symlink Race Exploit in Zig

Educational exploit of the TOCTOU (Time Of Check, Time Of Use) race condition behind CVE-2021-25741, a vulnerability in the Kubernetes kubelet.

The project demonstrates locally (without a K8s cluster) how a concurrent symlink swap can bypass a path check between lstat() and open(), and proves that openat2() with RESOLVE_* flags is the fix.

Table of Contents

• The Problem
• How the Exploit Works
• Code Architecture
• Prerequisites
• Build
• Usage
  • Local test with Docker
  • Comparison openat vs openat2
  • Playing with the TOCTOU delay
  • Testing on a real Kubernetes node
• Expected Results
• Code Explanation
• Linux Syscalls Used
• Debug with strace
• References

---

The Problem

Path resolution under Linux

When the kernel resolves a path like /a/b/c/file, it does so component by component. At each step, if the component is a symlink, the kernel follows it automatically. This resolution is not atomic — the filesystem can change between two steps.

TOCTOU in the kubelet

The Kubernetes kubelet was doing exactly this vulnerable pattern:

1. CHECK : lstat(subPath) → "it's a directory, it's safe"
     ↕ RACE WINDOW — a process in the container swaps the directory for a symlink
2. USE   : mount(subPath) → follows the symlink, mounts the host filesystem

Between the check and the mount, a malicious process inside the container could replace the subPath with a symlink to the host /, thus gaining full access to the host filesystem.

The fix: openat2(2)

The openat2 syscall (kernel 5.6+) resolves the path and opens the file atomically, with constraints:

Flag	Effect
RESOLVE_NO_SYMLINKS	Refuses to follow any symlink → returns ELOOP
RESOLVE_BENEATH	Refuses to escape the base directory → returns EXDEV
RESOLVE_IN_ROOT	Treats the dirfd as the filesystem root
RESOLVE_NO_XDEV	Refuses to traverse mount points

With openat2, there is no TOCTOU window: if a symlink appears during resolution, the syscall fails immediately.

---

How the Exploit Works

Two threads cooperate to exploit the TOCTOU window:

workdir/
├── legit_dir/
│   └── secret.txt  → contains "LEGIT"
├── symlink_target/
│   └── secret.txt  → contains "PWNED"
└── target/          → swapped between real dir and symlink

Racer Thread

Runs in an ultra-fast loop and atomically swaps target/ between two states via renameat2(RENAME_EXCHANGE):

• State A: target/ is a real directory (contains secret.txt = "LEGIT")
• State B: target/ is a symlink to symlink_target/ (contains secret.txt = "PWNED")

One syscall per swap = maximum race window.

Victim Thread (simulates the kubelet)

Reproduces the vulnerable kubelet pattern:

1. fstatat("target", AT_SYMLINK_NOFOLLOW) — checks that it's a directory
2. Optional pause (simulates kubelet latency between check and use)
3. openat(dirfd, "target/secret.txt", O_RDONLY) — opens the file
4. read() — reads the content
5. If content == "PWNED" → race win (the symlink was followed)
6. If content == "LEGIT" → race loss (it was indeed the real directory)

In protected mode (--use-openat2), step 3 uses openat2 with RESOLVE_NO_SYMLINKS | RESOLVE_BENEATH. If a symlink is present, the kernel returns ELOOP instead of following it.

---

Code Architecture

src/
├── main.zig       # Entry point: parses CLI, creates shared state, spawns threads,
│                  # measures time, displays results
│
├── racer.zig      # Racer thread: prepares initial state (target → symlink),
│                  # then loops on renameat2(RENAME_EXCHANGE) to swap
│                  # target/ and legit_dir/ continuously
│
├── victim.zig     # Victim thread: loops N iterations of fstatat → delay →
│                  # openat/openat2 → read → compare "LEGIT" vs "PWNED"
│
├── setup.zig      # Creates the test tree: workdir/, legit_dir/,
│                  # symlink_target/, target/ with sentinel files
│
├── syscalls.zig   # Constants and wrappers for raw syscalls:
│                  # - open_how struct (kernel UAPI, 3×u64)
│                  # - RESOLVE_* flags
│                  # - RENAME_EXCHANGE (= 2)
│                  # - openat2() via linux.syscall4(.openat2, ...)
│                  # - rename_exchange() via linux.renameat2()
│                  # - Helpers : is_err(), to_errno(), to_fd()
│
└── stats.zig      # Lock-free atomic counters (std.atomic.Value(u64))
                   # for wins, losses, errors, eloop + formatted output

All syscalls are called via std.os.linux.* directly (no std.fs or std.posix wrappers). The only syscall without a wrapper in Zig 0.15 stdlib is openat2, which is called via linux.syscall4(.openat2, ...) with a manually defined open_how struct from the kernel UAPI headers.

---

Prerequisites

Tool	Version	Why
Zig	0.15.x	Compiler + cross-compilation
Docker	any	To run the Linux binary on Mac
colima	any	Docker runtime on macOS (or Docker Desktop)

The binary is compiled as static (musl libc) and runs on any Linux without dependencies.

Minimum kernel:

• 3.15+ for renameat2(RENAME_EXCHANGE)
• 5.6+ for openat2 with RESOLVE_* (needed only for --use-openat2)

Docker Desktop and colima on Mac use a 6.x kernel — everything is supported.

---

Build

Cross-compilation Mac → Linux

# ARM64 (Mac M1/M2/M3 → Docker colima / EC2 ARM)
zig build -Dtarget=aarch64-linux-musl -Doptimize=ReleaseSafe

# x86_64 (for x86 nodes)
zig build -Dtarget=x86_64-linux-musl -Doptimize=ReleaseSafe

The binary is in zig-out/bin/race-exploit.

$ file zig-out/bin/race-exploit
ELF 64-bit LSB executable, ARM aarch64, statically linked

---

Usage

Local test with Docker

# Start the Docker runtime (if macOS)
colima start

# Run the exploit (default vulnerable mode)
docker run --rm -v $(pwd)/zig-out/bin:/app alpine /app/race-exploit --iterations 10000

Comparison openat vs openat2

# Vulnerable mode (openat) — race works
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  /app/race-exploit --iterations 10000

# Protected mode (openat2) — race is blocked
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  /app/race-exploit --iterations 10000 --use-openat2

Or using the script that runs both:

./scripts/run_in_docker.sh --iterations 10000

Playing with the TOCTOU delay

The --delay-us parameter adds a delay between lstat (check) and openat (use). The longer the delay, the larger the TOCTOU window, and the higher the win rate:

# No delay — win rate ~25%
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  /app/race-exploit --iterations 50000

# 10µs — win rate ~35%
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  /app/race-exploit --iterations 50000 --delay-us 10

# 100µs — win rate ~50%
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  /app/race-exploit --iterations 50000 --delay-us 100

# 1000µs (1ms) — win rate ~50% (capped, racer swaps much faster)
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  /app/race-exploit --iterations 10000 --delay-us 1000

In a real kubelet, the latency between the subPath check and the bind mount is on the order of several milliseconds (API calls, mount namespace preparation, etc.), making the race very reliable in real conditions.

Full CLI Options