teamos-hub/regresshionThis is a POC I wrote for CVE-2024-6387
Qualys Security Advisory
regreSSHion: RCE in OpenSSH's server, on glibc-based Linux systems (CVE-2024-6387)
Summary SSH-2.0-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3 (Debian 3.0r6, from 2005)
All it takes is a leap of faith
-- The Interrupters, "Leap of Faith"
Preliminary note: OpenSSH is one of the most secure software in the world; this vulnerability is one slip-up in an otherwise near-flawless implementation. Its defense-in-depth design and code are a model and an inspiration, and we thank OpenSSH's developers for their exemplary work.
We discovered a vulnerability (a signal handler race condition) in OpenSSH's server (sshd): if a client does not authenticate within LoginGraceTime seconds (120 by default, 600 in old OpenSSH versions), then sshd's SIGALRM handler is called asynchronously, but this signal handler calls various functions that are not async-signal-safe (for example, syslog()). This race condition affects sshd in its default configuration.
On investigation, we realized that this vulnerability is in fact a regression of CVE-2006-5051 ("Signal handler race condition in OpenSSH before 4.4 allows remote attackers to cause a denial of service (crash), and possibly execute arbitrary code"), which was reported in 2006 by Mark Dowd.
This regression was introduced in October 2020 (OpenSSH 8.5p1) by commit 752250c ("revised log infrastructure for OpenSSH"), which accidentally removed an "#ifdef DO_LOG_SAFE_IN_SIGHAND" from sigdie(), a function that is directly called by sshd's SIGALRM handler. In other words:
OpenSSH < 4.4p1 is vulnerable to this signal handler race condition, if not backport-patched against CVE-2006-5051, or not patched against CVE-2008-4109, which was an incorrect fix for CVE-2006-5051;
4.4p1 <= OpenSSH < 8.5p1 is not vulnerable to this signal handler race condition (because the "#ifdef DO_LOG_SAFE_IN_SIGHAND" that was added to sigdie() by the patch for CVE-2006-5051 transformed this unsafe function into a safe _exit(1) call);
8.5p1 <= OpenSSH < 9.8p1 is vulnerable again to this signal handler race condition (because the "#ifdef DO_LOG_SAFE_IN_SIGHAND" was accidentally removed from sigdie()).
This vulnerability is exploitable remotely on glibc-based Linux systems, where syslog() itself calls async-signal-unsafe functions (for example, malloc() and free()): an unauthenticated remote code execution as root, because it affects sshd's privileged code, which is not sandboxed and runs with full privileges. We have not investigated any other libc or operating system; but OpenBSD is notably not vulnerable, because its SIGALRM handler calls syslog_r(), an async-signal-safer version of syslog() that was invented by OpenBSD in 2001.
To exploit this vulnerability remotely (to the best of our knowledge, CVE-2006-5051 has never been successfully exploited before), we drew inspiration from a visionary paper, "Delivering Signals for Fun and Profit", which was published in 2001 by Michal Zalewski:
https://lcamtuf.coredump.cx/signals.txt
Nevertheless, we immediately faced three major problems:
From a theoretical point of view, we must find a useful code path that, if interrupted at the right time by SIGALRM, leaves sshd in an inconsistent state, and we must then exploit this inconsistent state inside the SIGALRM handler.
From a practical point of view, we must find a way to reach this useful code path in sshd, and maximize our chances of interrupting it at the right time.
From a timing point of view, we must find a way to further increase our chances of interrupting this useful code path at the right time, remotely.
To focus on these three problems without having to immediately fight against all the modern operating system protections (in particular, ASLR and NX), we decided to exploit old OpenSSH versions first, on i386, and then, based on this experience, recent versions:
First, "SSH-2.0-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3", from "debian-30r6-dvd-i386-binary-1_NONUS.iso": this is the first Debian version that has privilege separation enabled by default and that is patched against all the critical vulnerabilities of that era (in particular, CVE-2003-0693 and CVE-2002-0640).
To remotely exploit this version, we interrupt a call to free() with SIGALRM (inside sshd's public-key parsing code), leave the heap in an inconsistent state, and exploit this inconsistent state during another call to free(), inside the SIGALRM handler.
In our experiments, it takes ~10,000 tries on average to win this race condition; i.e., with 10 connections (MaxStartups) accepted per 600 seconds (LoginGraceTime), it takes ~1 week on average to obtain a remote root shell.
Second, "SSH-2.0-OpenSSH_4.2p1 Debian-7ubuntu3", from "ubuntu-6.06.1-server-i386.iso": this is the last Ubuntu version that is still vulnerable to CVE-2006-5051 ("Signal handler race condition in OpenSSH before 4.4").
To remotely exploit this version, we interrupt a call to pam_start() with SIGALRM, leave one of PAM's structures in an inconsistent state, and exploit this inconsistent state during a call to pam_end(), inside the SIGALRM handler.
In our experiments, it takes ~10,000 tries on average to win this race condition; i.e., with 10 connections (MaxStartups) accepted per 120 seconds (LoginGraceTime), it takes ~1-2 days on average to obtain a remote root shell.
Finally, "SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2", from "debian-12.5.0-i386-DVD-1.iso": this is the current Debian stable version, and it is vulnerable to the regression of CVE-2006-5051.
To remotely exploit this version, we interrupt a call to malloc() with SIGALRM (inside sshd's public-key parsing code), leave the heap in an inconsistent state, and exploit this inconsistent state during another call to malloc(), inside the SIGALRM handler (more precisely, inside syslog()).
In our experiments, it takes ~10,000 tries on average to win this race condition, so ~3-4 hours with 100 connections (MaxStartups) accepted per 120 seconds (LoginGraceTime). Ultimately, it takes ~6-8 hours on average to obtain a remote root shell, because we can only guess the glibc's address correctly half of the time (because of ASLR).
This research is still a work in progress:
we have targeted virtual machines only, not bare-metal servers, on a mostly stable network link (~10ms packet jitter);
we are convinced that various aspects of our exploits can be greatly improved;
we have started to work on an amd64 exploit, which is much harder because of the stronger ASLR.
A few days after we started our work on amd64, we noticed the following bug report (in OpenSSH's public Bugzilla), about a deadlock in sshd's SIGALRM handler:
https://bugzilla.mindrot.org/show_bug.cgi?id=3690
We therefore decided to contact OpenSSH's developers immediately (to let them know that this deadlock is caused by an exploitable vulnerability), we put our amd64 work on hold, and we started to write this advisory.
But that's not like me, I'm breaking free
-- The Interrupters, "Haven't Seen the Last of Me"
The SIGALRM handler of this OpenSSH version calls packet_close(), which calls buffer_free(), which calls xfree() and hence free(), which is not async-signal-safe:
Consequently, we started to read the malloc code of this Debian's glibc (2.2.5), to see if a first call to free() can be interrupted by SIGALRM and exploited during a second call to free() inside the SIGALRM handler (at lines 341-344, above). Because this glibc's malloc is not hardened against the unlink() technique pioneered by Solar Designer in 2000, we quickly spotted an interesting code path in chunk_free() (which is called internally by free()):
To exploit this code path, we arrange for sshd's heap to have the following layout (chunk_X, chunk_Y, and chunk_Z are malloc()ated chunks of memory, and p, s, f, b are their prev_size, size, fd, and bk fields):
-----|---+---------------|---+---------------|---+---------------|----- ... |p|s|f|b| chunk_X |p|s|f|b| chunk_Y |p|s|f|b| chunk_Z | ... -----|---+---------------|---+---------------|---+---------------|----- |<------------->| user data
First, if a call to free(chunk_Y) is interrupted by SIGALRM after line 3246 but before line 3251, then chunk_Y is already marked as free (because chunk_Z's PREV_INUSE bit is cleared at line 3246) but it is not yet linked into its doubly-linked list (at line 3251): in other words, chunk_Y's fd and bk pointers still contain user data (attacker- controlled data).
Second, if (inside the SIGALRM handler) packet_close() calls free(chunk_X), then the code block at lines 3230-3244 is entered (because chunk_Y is marked as free) and chunk_Y is unlink()ed (at line 3241): a so-called aa4bmo primitive (almost arbitrary 4 bytes mirrored overwrite), because chunk_Y's fd and bk pointers are still attacker- controlled. For more information on the unlink() technique and the aa4bmo primitive:
https://www.openwall.com/articles/JPEG-COM-Marker-Vulnerability#exploit http://phrack.org/issues/61/6.html#article
Last, with this aa4bmo primitive we overwrite the glibc's __free_hook function pointer (this old Debian version does not have ASLR, nor NX) with the address of our shellcode in the heap, thus achieving remote code execution during the next call to free() in packet_close().
Now they're taking over and they got complete control
-- The Interrupters, "Liberty"
To mount this attack against sshd, we interrupt a call to free() inside sshd's parsing code of a DSA public key (i.e., line 144 below is our free(chunk_Y)) and exploit it during one of the free() calls in packet_close() (i.e., one of the lines 341-344 above is our free(chunk_X)):
Initially, however, we were never able to win this race condition (i.e., interrupt the free() call at line 144 at the right time). Eventually, we realized that we could greatly improve our chances of winning this race: the DSA public-key parsing code allows us to call free() four times (at lines 704-707 below), and furthermore sshd allows us to attempt six user authentications (AUTH_FAIL_MAX); if any one of these 24 free() calls is interrupted at the right time, then we later achieve remote code execution inside the SIGALRM handler.
With this improvement, we finally won the race condition after ~1 month: we were happy (and did a root-shell dance), but we also felt that there was still room for improvement.
Don't worry, just wait and see
-- The Interrupters, "Haven't Seen the Last of Me"
We therefore implemented the following threefold timing strategy:
We do not wait until the last moment to send our (rather large) DSA public-key packet to sshd: instead, we send the entire packet minus one byte (the last byte) long before the LoginGraceTime, and send the very last byte at the very last moment, to minimize the effects of network delays. (And we disable the Nagle algorithm.)
We keep track of the median round-trip time (by regularly sending packets that produce a response from sshd), and keep track of the difference between the moment we are expecting our connection to be closed by sshd (essentially the moment we receive the first byte of sshd's banner, plus LoginGraceTime) and the moment our connection is really closed by sshd, and accordingly adjust our timing (i.e., the moment when we send the last byte of our DSA packet).
These time differences allow us to track clock skews and network delays, which show predictable patterns over time: we experimented with linear and spline regressions, but in the end, nothing worked better than simply re-using the most recent measurement. Possibly, deep learning might yield even better results; this is left as an exercise for the interested reader.
More importantly, we further increase our chances of winning this race condition by slowly adjusting our timing through involuntary feedback from sshd:
if we receive a response (SSH2_MSG_USERAUTH_FAILURE) to our DSA public-key packet, then we sent it too early (sshd had the time to receive our packet in the unprivileged child, parse it, send it to the privileged child, parse it there, and send a response all the way back to us);
if we cannot even send the last byte of our DSA packet, then we waited too long (sshd already received the SIGALRM and closed our connection);
if we can send the last byte of our DSA packet, and receive no response before sshd closes our connection, then our timing was reasonably accurate.
This feedback allows us to target what we call the "large" race window: hitting it does not guarantee that we win the race condition, but inside this large window are the 24 "small" race windows (inside the 24 free() calls) that, if hit, guarantee that we do win the race condition.
With these improvements, it takes ~10,000 tries on average to win this race condition; i.e., with 10 connections (MaxStartups) accepted per 600 seconds (LoginGraceTime), it takes ~1 week on average to obtain a remote root shell.
I sleep when the sun starts to rise
-- The Interrupters, "Alien"
The SIGALRM handler of this OpenSSH version does not call packet_close() anymore; moreover, this Ubuntu's glibc (2.3.6) always takes a mandatory lock when entering the functions of the malloc family (even if single- threaded like sshd), which prevents us from interrupting a call to one of the malloc functions and later exploiting it during another call to these functions (they would always deadlock). We must find another solution.
CVE-2006-5051 mentions a double-free in GSSAPI, but GSSAPI (or Kerberos) is not enabled by default, so this does not sound very appealing. On the other hand, PAM is enabled by default, and pam_end() is called by sshd's SIGALRM handler (and is, of course, not async-signal-safe). We therefore searched for a PAM function that, if interrupted by SIGALRM at the right time, would leave PAM's internal structures in an inconsistent state, exploitable during pam_end() in the SIGALRM handler. We found pam_set_data():
If this function is interrupted by SIGALRM after line 66 but before line 74, then data_entry is already linked into PAM's structures (pamh), but its cleanup field (a function pointer) is not yet initialized (since the malloc() at line 57 does not initialize its memory). If we are able to control cleanup (through leftovers from previous heap allocations), then we can execute arbitrary code when pam_end() (inside the SIGALRM handler) calls _pam_free_data() (at line 118):
This would have been an extremely simple exploit; unfortunately, we completely overlooked that pam_set_data() can only be called from PAM modules: if we interrupt it with SIGALRM, then pamh->caller_is is still _PAM_CALLED_FROM_MODULE, in which case pam_end() returns immediately, without ever calling _pam_free_data(). Back to the drawing board.
Not giving up, it's not what we do
-- The Interrupters, "Title Holder"
We noticed that, at line 601 below, sshd passes a pointer to its global sshpam_handle pointer directly to pam_start() (which is called once per connection):
We therefore decided to look into pam_start() itself: if interrupted by SIGALRM, it might leave the structure pointed to by sshpam_handle in an inconsistent state, which could then be exploited inside the SIGALRM handler, when "pam_end(sshpam_handle, sshpam_err)" is called.
At line 32, pam_start() immediately sets sshd's sshpam_handle to a calloc()ated chunk of memory; this is safe, because calloc() initializes this memory to zero. On the other hand, if _pam_add_handler() (which is called multiple times by pam_start()) is interrupted by SIGALRM after line 874 but before line 886, then a malloc()ated structure is linked into pamh, but its next field is not yet initialized. If we are able to control next (through leftovers from previous heap allocations), then we can pass an arbitrary pointer to free() during the call to pam_end() (inside the SIGALRM handler), at line 1020 (and line 1017) below:
Because the malloc of this Ubuntu's glibc is already hardened against the old unlink() technique, we decided to transform our arbitrary free() into the Malloc Maleficarum's House of Mind (fastbin version): we free() our own NON_MAIN_ARENA chunk, point our fake arena to sshd's .got.plt (this Ubuntu's sshd has ASLR but not PIE), and overwrite _exit()'s entry with the address of our shellcode in the heap (this Ubuntu's heap is still executable by default). For more information on the Malloc Maleficarum:
https://seclists.org/bugtraq/2005/Oct/118
I learned everything the hard way
-- The Interrupters, "The Hard Way"
To mount this attack against sshd, we initially faced three problems:
The House of Mind requires us to store the pointer to our fake arena at address 0x08100000 in the heap; but are we able to store attacker- controlled data at such a high address? Because sshd calls pam_start() at the very beginning of the user authentication, we do not control anything except the user name itself; luckily, a user name of length ~128KB (shorter than DEFAULT_MMAP_THRESHOLD) allows us to store our own data at address 0x08100000.
The size field of our fake NON_MAIN_ARENA chunk must not be too large (to pass free()'s security checks); i.e., it must contain null bytes. But our long user name is a null-terminated string that cannot contain null bytes; luckily we remembered that _pam_free_handlers_aux() zeroes the structures that it free()s (line 1019 above): we therefore "patch" the size field of our fake chunk with such a memset(0), and only then free() it.
We must survive several calls to free() (at lines 1017 and 1020 above) before the free() of our fake NON_MAIN_ARENA chunk. We transform these free()s into no-ops by pointing them to fake IS_MMAPPED chunks: free() calls munmap_chunk(), which calls munmap(), which fails because these fake IS_MMAPPED chunks are misaligned; effectively a no-op, because assert()ion failures are not enforced in this Ubuntu's glibc.
Finally, our long user name also allows us to control the potentially uninitialized next field of 20 different structures (through leftovers from temporary copies of our long user name), because pam_start() calls _pam_add_handler() multiple times; i.e., our large race window contains 20 small race windows.
Same tricks they used before
-- The Interrupters, "Divide Us"
For this attack against Ubuntu 6.06.1, we simply re-used the timing strategy that we used against Debian 3.0r6: it takes ~10,000 tries on average to win the race condition, and with 10 connections (MaxStartups) accepted per 120 seconds (LoginGraceTime), it takes ~1-2 days on average to obtain a remote root shell.
Note: because this Ubuntu's glibc always takes a mandatory lock when entering the functions of the malloc family, an unlucky attacker might deadlock all 10 MaxStartups connections before obtaining a root shell; we have not tried to work around this problem because our ultimate goal was to exploit a modern OpenSSH version anyway.
Now you're ready, take the demons head on
-- The Interrupters, "Be Gone"
The SIGALRM handler of this OpenSSH version does not call packet_close() nor pam_end(); in fact it calls only one interesting function, syslog():
Our two key questions, then, are: Does the syslog() of this Debian's glibc (2.36) call async-signal-unsafe functions such as malloc() and free()? And if yes, does this glibc still take a mandatory lock when entering the functions of the malloc family?
Note: because we do not control anything about these malloc()ations (not their order, not their sizes, not their contents), we took the "rce" at line 166 as a much-needed good omen.
And luckily for us, the answer to our second question is no; since October 2017, the glibc's malloc functions do not take any lock anymore, when single-threaded (like sshd):
https://sourceware.org/git?p=glibc.git;a=commit;h=a15d53e2de4c7d83bda251469d92a3c7b49a90db https://sourceware.org/git?p=glibc.git;a=commit;h=3f6bb8a32e5f5efd78ac08c41e623651cc242a89 https://sourceware.org/git?p=glibc.git;a=commit;h=905a7725e9157ea522d8ab97b4c8b96aeb23df54
Moreover, this Debian version suffers from the ASLR weakness described in the following great blog posts (by Justin Miller and Mathias Krause, respectively):
https://zolutal.github.io/aslrnt/ https://grsecurity.net/toolchain_necromancy_past_mistakes_haunting_aslr
Concretely, in the case of sshd on i386, every memory mapping is randomized normally (sshd's PIE, the heap, most libraries, the stack), but the glibc itself is always mapped either at address 0xb7200000 or at address 0xb7400000; in other words, we can correctly guess the glibc's address half of the time (a small price to pay for defeating ASLR). In our exploit we assume that the glibc is mapped at address 0xb7400000, because it is slightly more common than 0xb7200000.
Our next question is: which code paths inside the glibc's malloc functions, if interrupted by SIGALRM at the right time, leave the heap in an inconsistent state, exploitable during one of the malloc() calls inside the SIGALRM handler?
We found several interesting (and surprising!) code paths, but the one we chose involves only relative sizes, not absolute addresses (unlike various code paths inside unlink_chunk(), for example); this difference might prove crucial for a future amd64 exploit. This code path, inside malloc(), splits a large free chunk (victim) into two smaller chunks; the first chunk is returned to malloc()'s caller (at line 4345) and the second chunk (remainder) is linked into an unsorted list of free chunks (at lines 4324-4327):
If this code path is interrupted by SIGALRM after line 4327 but before line 4339, then the remainder chunk of this split is already linked into the unsorted list of free chunks (lines 4324-4327), but its size field (mchunk_size) is not yet initialized (line 4339).
If we are able to control its size field (through leftovers from previous heap allocations), then we can make this remainder chunk larger and overlap with other heap chunks, and therefore corrupt heap memory when this enlarged, overlapping remainder chunk is eventually malloc()ated and written to (inside the SIGALRM handler).
Our last question, then, is: given that we do not control anything about the malloc() calls inside the SIGALRM handler, what can we overwrite in the heap to achieve arbitrary code execution before sshd calls _exit() (in sshsigdie())?
Because __tzfile_read() (inside the SIGALRM handler) malloc()ates a FILE structure in the heap (at line 166 above), and because FILE structures have a long history of abuse for arbitrary code execution, we decided to aim our heap corruption at this FILE structure. This is, however, easier said than done: our heap corruption is very limited, and FILE structures have been significantly hardened over the years (by IO_validate_vtable() and PTR_DEMANGLE(), for example).
Eventually, we devised the following technique (which seems to be specific to the i386 glibc -- the amd64 glibc does not seem to use _vtable_offset at all):
with our limited heap corruption, we overwrite the _vtable_offset field (a single signed char) of __tzfile_read()'s FILE structure;
the glibc's libio functions will therefore look for this FILE structure's vtable pointer (a pointer to an array of function pointers) at a non-zero offset (our overwritten _vtable_offset), instead of the default zero offset;
we (attackers) can easily control this fake vtable pointer (through leftovers from previous heap allocations), because the FILE structure around this offset is not explicitly initialized by fopen();
to pass the glibc's security checks, our fake vtable pointer must point somewhere into the __libc_IO_vtables section: we decided to point it to the vtable for wide-character streams, _IO_wfile_jumps (i.e., to 0xb761b740, since we assume that the glibc is mapped at address 0xb7400000);
as a result, __fread_unlocked() (at line 186 above) calls _IO_wfile_underflow() (instead of _IO_file_underflow()), which calls a function pointer (__fct) that basically comes from a structure whose pointer (_codecvt) is yet another field of the FILE structure;
we (attackers) can easily control this _codecvt pointer (through leftovers from previous heap allocations, because this field of the FILE structure is not explicitly initialized by fopen()), which also allows us to control the __fct function pointer.
In summary, by overwriting a single byte (_vtable_offset) of the FILE structure malloc()ated by fopen(), we can call our own __fct function pointer and execute arbitrary code during __fread_unlocked().
I wanted it perfect, no wrinkles in it
-- The Interrupters, "In the Mirror"
To mount this attack against sshd's privileged child, let us first imagine the following heap layout (the "XXX"s are "barrier" chunks that allow us to make holes in the heap; for example, small memory-leaked chunks):
---|----------------------------------------------|---|------------|---
| XXX | large hole | XXX | small hole | XXX |
|---|---|---|---|---|
| ~8KB | 320B |
---|-----------------------|----------------------|---|------------|---
| XXX | large allocated chunk | free remainder chunk | XXX | small hole | XXX |
|---|---|---|---|---|---|
| ~4KB | ~4KB | 320B |
---|-----------------------|----------------------|---|------------|---
| XXX | large allocated chunk | real remainder chunk | XXX | small hole | XXX |
|---|---|---|---|---|---|
| ~4KB | <-------------------------------------> |
artificially enlarged remainder chunk
---|-----------------------|----------------------|---|------------|--- XXX| large allocated chunk | |XXX| FILE |XXX ---|-----------------------|----------------------|---|--|---------|--- | ~4KB |<--------------------------->|<------->| 4KB read buffer remainder
we therefore overwrite parts of the FILE structure with the internal header of this small remainder chunk: more precisely, we overwrite the FILE's _vtable_offset with the third byte of this header's bk field, which is a pointer to the unsorted list of free chunks, 0xb761d7f8 (i.e., we overwrite _vtable_offset with 0x61);
then, as explained in the "Theory" subsection, __fread_unlocked() calls _IO_wfile_underflow() (instead of _IO_file_underflow()), which calls our own __fct function pointer (through our own _codecvt pointer) and executes our arbitrary code.
Note: we have not yet explained how to reliably go from a controlled _codecvt pointer to a controlled __fct function pointer; we will do so, but we must first solve a more pressing problem.
Indeed, we learned from our work on older OpenSSH versions that we will never win this signal handler race condition if our large race window contains only one small race window. Consequently, we implemented the following strategy, based on the following heap layout:
---|------------|---|------------|---|------------|---|------------|---
| XXX | large hole 1 | XXX | small hole 1 | XXX | large hole 2 | XXX | small hole 2 | ... |
|---|---|---|---|---|---|---|---|---|
The last packet that we send to sshd (shortly before the delivery of SIGALRM) forces sshd to perform the following sequence of malloc() calls: malloc(~4KB), malloc(304), malloc(~4KB), malloc(304), etc.
1/ Our first malloc(~4KB) splits the large hole 1 in two:
if this first split is interrupted by SIGALRM at the right time, then the fopen() inside the SIGALRM handler malloc()ates the small hole 1 for its FILE structure, and we achieve arbitrary code execution as explained above;
if not, then we malloc()ate the small hole 1 ourselves with our first malloc(304), and:
2/ Our second malloc(~4KB) splits the large hole 2 in two:
if this second split is interrupted by SIGALRM at the right time, then the fopen() inside the SIGALRM handler malloc()ates the small hole 2 for its FILE structure, and we achieve arbitrary code execution as explained above;
if not, then we malloc()ate the small hole 2 ourselves with our second malloc(304), etc.
We were able to make 27 pairs of such large and small holes in sshd's heap (28 would exceed PACKET_MAX_SIZE, 256KB): our large race window now contains 27 small race windows! Achieving this complex heap layout was extremely painful and time-consuming, but the two highlights are:
To reliably achieve this heap layout, we send five different public-key packets to sshd (packets a/ to d/ can be sent long before SIGALRM; most of packet e/ can also be sent long before SIGALRM, but its very last byte must be sent at the very last moment):
a/ We malloc()ate and free() a variety of tcache chunks, to ensure that the heap allocations that we do not control end up in these tcache chunks and do not interfere with our careful heap layout.
b/ We malloc()ate and free() chunks of various sizes, to make our 27 pairs of large and small holes (and the corresponding "barrier" chunks).
c/ We malloc()ate and free() ~4KB chunks and 320B chunks, to:
write the fake header (the large size field) of our potentially enlarged remainder chunk, into the middle of our large holes;
write the fake footer of our potentially enlarged remainder chunk, to the end of our small holes (to pass the glibc's security checks);
write our fake vtable and _codecvt pointers, into our small holes (which are potential FILE structures).
d/ We malloc()ate and free() one very large string (nearly 256KB), to ensure that our large and small holes are removed from the unsorted list of free chunks and placed into their respective malloc bins.
e/ We force sshd to perform our final sequence of malloc() calls (malloc(~4KB), malloc(304), malloc(~4KB), malloc(304), etc), to open our 27 small race windows.
Attentive readers may have noticed that we have still not addressed (literally and figuratively) the problem of _codecvt. In fact, _codecvt is a pointer to a structure (_IO_codecvt) that contains a pointer to a structure (__gconv_step) that contains the __fct function pointer that allows us to execute arbitrary code. To reliably control __fct through _codecvt, we simply point _codecvt to one of the glibc's malloc bins, which conveniently contains a pointer to one of our free chunks in the heap, which contains our own __fct function pointer to arbitrary glibc code (all of these glibc addresses are known to us, because we assume that the glibc is mapped at address 0xb7400000).
We're running out of time
-- The Interrupters, "As We Live"
As we implemented this third exploit, it became clear that we could not simply re-use the timing strategy that we had used against the two older OpenSSH versions: we were never winning this new race condition. Eventually, we understood why:
It takes a long time (~10ms) for sshd to parse our fifth and last public key (packet e/ above); in other words, our large race window is too large (our 27 small race windows are like needles in a haystack).
The user_specific_delay() that was introduced recently (OpenSSH 7.8p1) delays sshd's response to our last public-key packet by up to ~9ms and therefore destroys our feedback-based timing strategy.
As a result, we developed a completely different timing strategy:
from time to time, we send our last public-key packet with a little mistake that produces an error response (lines 138-142 below), right before the call to sshkey_from_blob() that parses our public key;
from time to time, we send our last public-key packet with another little mistake that produces an error response (lines 151-155 below), right after the call to sshkey_from_blob() that parses our public key;
the difference between these two response times is the time that it takes for sshd to parse our last public key, and this allows us to precisely time the transmission of our last packets (to ensure that sshd has the time to parse our public key in the unprivileged child, send it to the privileged child, and start to parse it there, before the delivery of SIGALRM).
With this change in strategy, it takes ~10,000 tries on average to win the race condition; i.e., with 100 connections (MaxStartups) accepted per 120 seconds (LoginGraceTime), it takes ~3-4 hours on average to win the race condition, and ~6-8 hours to obtain a remote root shell (because of ASLR).
What's your plan for tomorrow?
-- The Interrupters, "Take Back the Power"
We decided to target Rocky Linux 9 (a Red Hat Enterprise Linux 9 derivative), from "Rocky-9.4-x86_64-minimal.iso", for two reasons:
its OpenSSH version (8.7p1) is vulnerable to this signal handler race condition and its glibc is always mapped at a multiple of 2MB (because of the ASLR weakness discussed in the previous "Theory" subsection), which makes partial pointer overwrites much more powerful;
the syslog() function (which is async-signal-unsafe but is called by sshd's SIGALRM handler) of this glibc version (2.34) internally calls __open_memstream(), which malloc()ates a FILE structure in the heap, and also calls calloc(), realloc(), and free() (which gives us some much-needed freedom).
With a heap corruption as a primitive, two FILE structures malloc()ated in the heap, and 21 fixed bits in the glibc's addresses, we believe that this signal handler race condition is exploitable on amd64 (probably not in ~6-8 hours, but hopefully in less than a week). Only time will tell.
Side note: we discovered that Ubuntu 24.04 does not re-randomize the ASLR of its sshd children (it is randomized only once, at boot time); we tracked this down to the patch below, which turns off sshd's rexec_flag. This is generally a bad idea, but in the particular case of this signal handler race condition, it prevents sshd from being exploitable: the syslog() inside the SIGALRM handler does not call any of the malloc functions, because it is never the very first call to syslog().
https://git.launchpad.net/ubuntu/+source/openssh/tree/debian/patches/systemd-socket-activation.patch
The storm has come and gone
-- The Interrupters, "Good Things"
On June 6, 2024, this signal handler race condition was fixed by commit 81c1099 ("Add a facility to sshd(8) to penalise particular problematic client behaviours"), which moved the async-signal-unsafe code from sshd's SIGALRM handler to sshd's listener process, where it can be handled synchronously:
https://github.com/openssh/openssh-portable/commit/81c1099d22b81ebfd20a334ce986c4f753b0db29
Because this fix is part of a large commit (81c1099), on top of an even larger defense-in-depth commit (03e3de4, "Start the process of splitting sshd into separate binaries"), it might prove difficult to backport. In that case, the signal handler race condition itself can be fixed by removing or commenting out the async-signal-unsafe code from the sshsigdie() function; for example:
sshsigdie(const char *file, const char *func, int line, int showfunc, LogLevel level, const char *suffix, const char *fmt, ...) { #if 0 va_list args;
va_start(args, fmt);
sshlogv(file, func, line, showfunc, SYSLOG_LEVEL_FATAL,
suffix, fmt, args);
va_end(args);
Finally, if sshd cannot be updated or recompiled, this signal handler race condition can be fixed by simply setting LoginGraceTime to 0 in the configuration file. This makes sshd vulnerable to a denial of service (the exhaustion of all MaxStartups connections), but it makes it safe from the remote code execution presented in this advisory.
We thank OpenSSH's developers for their outstanding work and close collaboration on this release. We also thank the distros@openwall. Finally, we dedicate this advisory to Sophia d'Antoine.
2024-05-19: We contacted OpenSSH's developers. Successive iterations of patches and patch reviews followed.
2024-06-20: We contacted the distros@openwall.
2024-07-01: Coordinated Release Date.
| ~8KB |
| 320B |
| ~8KB |
| 320B |