Bypassing Spectre-BTI User Space Mitigations on Linux

This is a working document, please send us feedback if you think we got something wrong or if we missed a citation

Version 1.0

José Oliveira (esoj)

Rodrigo Branco (BSDaemon)

Introduction

When testing the success rate of Spectre-BTI attacks, we detected a strange pattern when using the kernel API as mitigation[^1]. Our tests revealed that the Linux kernel fails to correctly mitigate the attack leaving the process exposed for a short period of time after the syscall.

Further investigation showed that the kernel does not issue an IBPB immediately during the syscall. The ib_prctl_set[^2] function updates the Thread Information Flags (TIFs) for the task and updates the SPEC_CTRL MSR on the function __speculation_ctrl_update [^3], but the IBPB is only issued on the next schedule, when the TIF bits are checked. This leaves the victim vulnerable to values already injected on the BTB, prior to the prctl syscall. The behavior is only corrected after a reschedule of the taks happens. Furthermore, the kernel entrance (due to the syscall itself), does not issue an IBPB in the default scenarios (i.e., when the kernel protects itself via retpoline or eIBRS).

The prctl mitigation

Executing a prctl to mitigate spectre-BTI attacks using: prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_FORCE_DISABLE, 0, 0); leads to the ib_prctl_set[^2] function on kernel 5.15. When the option SPEC_DISABLE is used the TIF bit for task_set_spec_ib_disable is set task_update_spec_tif is called:

static int ib_prctl_set(struct task_struct *task, unsigned long ctrl)
[...]
case PR_SPEC_FORCE_DISABLE:
    /*
     * Indirect branch speculation is always allowed when
     * mitigation is force disabled.
     */
    if (spectre_v2_user_ibpb == SPECTRE_V2_USER_NONE &&
        spectre_v2_user_stibp == SPECTRE_V2_USER_NONE)
        return -EPERM;

    if (!is_spec_ib_user_controlled())
        return 0;

    task_set_spec_ib_disable(task);
    if (ctrl == PR_SPEC_FORCE_DISABLE)
        task_set_spec_ib_force_disable(task);
    task_update_spec_tif(task);
    break;

task_set_spec_ib_disable calls set_tsk_thread_flag(tsk, TIF_SPEC_FORCE_UPDATE); and if the target task is the current it calls speculation_ctrl_update_current();

static void task_update_spec_tif(struct task_struct *tsk)
{
	/* Force the update of the real TIF bits */
	set_tsk_thread_flag(tsk, TIF_SPEC_FORCE_UPDATE);

	/*
	 * Immediately update the speculation control MSRs for the current
	 * task, but for a non-current task delay setting the CPU
	 * mitigation until it is scheduled next.
	 *
	 * This can only happen for SECCOMP mitigation. For PRCTL it's
	 * always the current task.
	 */
	if (tsk == current)
		speculation_ctrl_update_current();
}

The speculation_ctrl_update_current after the speculation_ctrl_update wrapper executes __speculation_ctrl_update with tifp = ~tifp, here the update of the wrmsr for setting STIBP is executed but no IBPB is issued :

static __always_inline void __speculation_ctrl_update(unsigned long tifp,
						      unsigned long tifn)
{
	unsigned long tif_diff = tifp ^ tifn;
	u64 msr = x86_spec_ctrl_base;
	bool updmsr = false;

	lockdep_assert_irqs_disabled();

	/* Handle change of TIF_SSBD depending on the mitigation method. */
	if (static_cpu_has(X86_FEATURE_VIRT_SSBD)) {
		if (tif_diff & _TIF_SSBD)
			amd_set_ssb_virt_state(tifn);
	} else if (static_cpu_has(X86_FEATURE_LS_CFG_SSBD)) {
		if (tif_diff & _TIF_SSBD)
			amd_set_core_ssb_state(tifn);
	} else if (static_cpu_has(X86_FEATURE_SPEC_CTRL_SSBD) ||
		   static_cpu_has(X86_FEATURE_AMD_SSBD)) {
		updmsr |= !!(tif_diff & _TIF_SSBD);
		msr |= ssbd_tif_to_spec_ctrl(tifn);
	}

	/* Only evaluate TIF_SPEC_IB if conditional STIBP is enabled. */
	if (IS_ENABLED(CONFIG_SMP) &&
	    static_branch_unlikely(&switch_to_cond_stibp)) {
		updmsr |= !!(tif_diff & _TIF_SPEC_IB);
		msr |= stibp_tif_to_spec_ctrl(tifn);
	}

	if (updmsr)
		wrmsrl(MSR_IA32_SPEC_CTRL, msr);
}

The seccomp syscall also uses ib_prctl_set[^2] as mitigation, inside arch_seccomp_spec_mitigate[^4] so the same result is expected with seccomp.

Tests

While via code analysis we are certain that the window for exploiting exists, it was unclear if it was big enough to have the victim loading secrets and the attacker leaking them (since secrets are expected to not be in the victim address space until the prctl call is issued). The tests were executed on a bare metal machine with support for hardware mitigations, with a ubuntu 22.04.1 LTS installed:

Kernel is Linux 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64
CPU is Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz
* Hardware support (CPU microcode) for mitigation techniques
  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates IBRS capability:  YES  (SPEC_CTRL feature bit)
  * Indirect Branch Prediction Barrier (IBPB)
    * CPU indicates IBPB capability:  YES  (SPEC_CTRL feature bit)
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates STIBP capability:  YES  (Intel STIBP feature bit)
  * Speculative Store Bypass Disable (SSBD)

The test code consists of two processes executing on the same logic core. The attacker constantly poisons the BTB with the address of a spectre gadget present on a victim process. The victim process measures the misprediction rate by checking if a test variable was accessed by the spectre gadget function. This usually returns the following output:

esoj@oxigenio:~/CPU_exploits/prctlbleed$ ./attacker  0x55555554123 0x55555555345 0 &
esoj@oxigenio:~/CPU_exploits/prctlbleed$ ./victim-PRCTL 0x55555554123 0x55555555345 0
Rate: 941/1000  
Rate: 1000/1000  
Rate: 999/1000  
Rate: 1000/1000  
Rate: 1000/1000  
Rate: 997/1000  
Rate: 994/1000  
Rate: 996/1000  
Rate: 998/1000  
Rate: 993/1000  
Total misspredict rate: 9918/10000 (99.18 %)

Then, PRCTL is used to mitigate the attack. The mitigation can be enabled by adding prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_INDIRECT_BRANCH, PR_SPEC_FORCE_DISABLE, 0, 0); on the begining of the program. This is expected to mitigate the spectre-BTI attack:

PRCTL GET value 0x9
Rate: 0/1000  
Rate: 0/1000  
Rate: 0/1000  
Rate: 0/1000  
Rate: 0/1000  
Rate: 0/1000  
Rate: 0/1000  
Rate: 0/1000  
Rate: 0/1000  
Rate: 0/1000  
Total misspredict rate: 0/10000 (0.00 %)

However, some of the tests showed a different result:

Rate: 50510/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Total misspredict rate: 50510/1000000 (5.05 %)

And changing the 'nice' (priority) seems to affect the misprediction rate:

esoj@oxigenio:~/CPU_exploits/prctlbleed$ sudo nice -n -19 ./victim-PRCTL 0x55555554123 0x55555555345 0
Rate: 99994/100000
Rate: 7716/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Total misspredict rate: 107710/1000000 (10.77 %)

esoj@oxigenio:~/CPU_exploits/prctlbleed$ sudo nice -n 19 ./victim-PRCTL 0x55555554123 0x55555555345 0
Rate: 16715/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Rate: 0/100000
Total misspredict rate: 16715/1000000 (1.67 %)

This indicates that the prctl only protected the process after the next schedule, as understood by code analysis. Another strange behavior of this test is that after an incorrect branch, the speculation path should be corrected and the true value must be written on the BTB. Since there are no other attackers on the sibling thread to re-poison the BTB, such high misprediction values are unexpected.