CVE-2023-5217: libvpx VP8 Encoding Heap Overflow PoC

CVE-2023-5217 is an in-the-wild exploited libvpx vulnerability that was found by Clément Lecigne of Google's Threat Analysis Group to be targeting Chrome.

This repo shows how to trigger CVE-2023-5217 in the browser using the WebCodecs and MediaRecorder APIs. CVE-2023-5217 allows for a heap buffer overflow with a controlled overflow length and an overwrite of a repeated small 4-byte value. It is not currently known how CVE-2023-5217 was exploited in the wild.

Around the time of public disclosure, there were two patches in libvpx and one in Chromium that remediated CVE-2023-5217. The libvpx patches included disabling VP8 thread number changes and a test for multithreaded encoding. The Chromium patch disabled adjusting the number of threads in WebCodecs.

Underlying Issue in libvpx v1.13.0 Ugly Duckling

Summary

libvpx is a library that handles VP8/VP9 encoding and decoding.

The key issue in CVE-2023-5217 is that reducing the number of threads while increasing the frame height in a libvpx VP8 encoding session causes a linear heap overflow of a controlled length and controlled overwrite of a repeated small 4-byte value. The difference in frame height controls the length of the overwrite, and the new frame width controls the 4-byte value that is repeatedly written. This vulnerability can be exploited multiple times to continuously write different small 4-byte values by reducing the height in each subsequent config.

Details

The libvpx VP8 encoder maintains an array called mt_current_mb_col that stores the current column being worked on by an encoder thread. This array is only allocated if there is more than one thread, and its size is a function of mb_rows, where mb_rows = frame_height >> 4 and the frame_height is rounded up to the nearest multiple of 16.

// https://github.com/webmproject/libvpx/blob/6512f994da13e2f27e6a7bd449efee0a374b55b7/vp8/common/alloccommon.c#L85
// The width and height are rounded up to a multiple of 16 and then assigned to `mb_rows` and `mb_cols`
int vp8_alloc_frame_buffers(VP8_COMMON *oci, int width, int height) {
    ...
    // Round up the width/height up to the nearest multiple of 16
    if ((width & 0xf) != 0) width += 16 - (width & 0xf);
    if ((height & 0xf) != 0) height += 16 - (height & 0xf);
    ...
    oci->mb_rows = height >> 4;
    oci->mb_cols = width >> 4;
    ...
}
// https://github.com/webmproject/libvpx/blob/6512f994da13e2f27e6a7bd449efee0a374b55b7/vp8/encoder/onyx_if.c#L1232
// This snippet shows the allocation of `mt_current_mb_col`
void vp8_alloc_compressor_data(VP8_COMP *cpi) {
  ...
  // Only allocate if we have more than 1 thread
  if (cpi->oxcf.multi_threaded > 1) {
    int i;

    vpx_free(cpi->mt_current_mb_col);
    // sizeof(*cpi->mt_current_mb_col) is 4
    CHECK_MEM_ERROR(&cpi->common.error, cpi->mt_current_mb_col,
                    vpx_malloc(sizeof(*cpi->mt_current_mb_col) * cm->mb_rows));
    for (i = 0; i < cm->mb_rows; ++i)
      vpx_atomic_init(&cpi->mt_current_mb_col[i], 0);
  }
  ...
}

Once libvpx finishes encoding a frame, it stores the number of columns encoded plus mt_sync_range into mt_current_mb_col.

// https://github.com/webmproject/libvpx/blob/6512f994da13e2f27e6a7bd449efee0a374b55b7/vp8/encoder/onyx_if.c#L1212
// Snippet where the mt_sync_range value is set, based on the width.
void vp8_alloc_compressor_data(VP8_COMP *cpi) {
    ...
#if CONFIG_MULTITHREAD
  if (width < 640) {
    cpi->mt_sync_range = 1;
  } else if (width <= 1280) {
    cpi->mt_sync_range = 4;
  } else if (width <= 2560) {
    cpi->mt_sync_range = 8;
  } else {
    cpi->mt_sync_range = 16;
  }
#endif
  ...
}
// https://github.com/webmproject/libvpx/blob/6512f994da13e2f27e6a7bd449efee0a374b55b7/vp8/encoder/encodeframe.c#L560
// Function where the attacker chosen value is written
static void encode_mb_row(...) {
  ...
  const int nsync = cpi->mt_sync_range; // This value is set in vp8_alloc_compressor_data
  vpx_atomic_int rightmost_col = VPX_ATOMIC_INIT(cm->mb_cols + nsync);
  ...
  if (vpx_atomic_load_acquire(&cpi->b_multi_threaded) != 0) {
    current_mb_col = &cpi->mt_current_mb_col[mb_row];
  }
  ...
  if (vpx_atomic_load_acquire(&cpi->b_multi_threaded) != 0) {
    // current_mb_col is a reference to mt_current_mb_col
    vpx_atomic_store_release(current_mb_col,
                             vpx_atomic_load_acquire(&rightmost_col));
  }
  ...
}

To overflow mt_current_mb_col, we need three encoding configurations:

1. config_init: During initialization, libvpx sets the initial_width and initial_height. These are the maximum possible bounds of later configurations. The number of threads here doesn't matter. We need the intermediate config because no subsequent width/height combinations can be larger than what we started with.
2. config_vuln: During reconfiguration, if more than one thread is used then libvpx creates the vulnerable mt_current_mb_col allocation based on config_vuln.height. This new height must be smaller than config_init.initial_height or else we'll get an error.
3. config_attack: During reconfiguration, if only one thread is used then libvpx will not reallocate mt_current_mb_col leaving it in a vulnerable state. libvpx will repeatedly write the value (config_attack.width >> 4) + 1 (where 1 is the variable mt_sync_range and the width is rounded up to the nearest multiple of 16) outside of the previously allocated bounds when the following condition holds:

\text{ceil}(\text{config}_{\text{init}}.\text{height}/16) \geq \text{ceil}(\text{config}_{\text{attack}}.\text{height}/16) \gt \text{ceil}(\text{config}_{\text{vuln}}.\text{height}/16)$$