StepJack: Benchmarking Computer-Use Agent Safety Against Multi-Step Indirect Prompt Injection

Setup

1. Python Environment Setup

First, install all dependencies and ensure you’re running Python 3.10.

conda create -n stepjack python=3.10
conda activate stepjack

pip install -r requirements.txt

2. Sandbox Environment Setup

We use the AWS environment provided by RedTeamCUA.

AWS

1. Configure NETWORK_INTERFACE_MAP:

   Configure the NETWORK_INTERFACE_MAP in aws_config.py based on your own AWS instances.

2. Configure your security group:

   It’s important to configure a security group that allows legitimate access to your hosted instances while blocking unauthorized access attempts. This is crucial because the AMI, by default, uses only basic password-based authentication, which is relatively easy to compromise.

   Your local machine will access the services hosted on AWS via HTTP/TCP connections. Therefore, you should explicitly allow access to the following ports: 8092, 6379, 9999, 5000, 9222, 8091, 2999, and 3000, which are used by either WebArena or TheAgentCompany.

   In addition, we use SSH to directly modify the underlying database in order to simulate injection scenarios. As such, SSH access from your local IP address must also be allowed in the security group settings.

3. Configure your AWS credentials by using aws configure, which will prompt you for:

   • AWS Access Key ID
   • AWS Secret Access Key
   • Default region name (Optional, you can press enter)

Experiment

scripts/run_gpt54_2step.sh shows one example of running GPT-5.4 CUA on StepJack.

1. Set Environment Variables

   CUA API keys

   # For GPT 
   unset OPENAI_BASE_URL
   export OPENAI_API_KEY="sk-proj-XXXX"
   
   # For Kimi
   export KIMI_API_KEY='sk-XXXX'
   
   # For Qwen
   export OPENAI_BASE_URL="https://dashscope-us.aliyuncs.com/compatible-mode/v1"
   export OPENAI_API_KEY="sk-XXXX"
   
   # For Claude
   export ANTHROPIC_API_KEY='sk-XXXX'
   

   AWS Credentials

   export AWS_REGION='XX'
   export AWS_ACCESS_KEY='XX'
   export AWS_SECRET_KEY='XX'
   

2. Run the Experiments

   # For GPT
   python run_gpt54_multienv.py \
      --observation_type screenshot \
      --agent_type GPT54Agent \
      --headless \
      --model gpt-5.4-mini-2026-03-17 \
      --result_dir ./results_gpt54 \
      --test_all_meta_path "./dataset/{subset}/{platform_instruction}/test_{k}_step.json" \
      --test_config_base_dir "./dataset/{subset}/{platform_instruction}"  \
      --max_steps {15*k} \
      --sleep_after_execution 4 \
      --provider_name aws \
      --num_envs 12 \
      --aws_ami {reddit | agentcompany}
   
   # For Kimi
   python run_kimi_multienv.py \
      --observation_type screenshot \
      --agent_type KimiAgent \
      --headless \
      --model Kimi-K2.5 \
      --result_dir ./results_kimi \
      --test_all_meta_path "./dataset/{subset}/{platform_instruction}/test_{k}_step.json" \
      --test_config_base_dir "./dataset/{subset}/{platform_instruction}"  \
      --max_steps {15*k} \
      --sleep_after_execution 4 \
      --provider_name aws \
      --num_envs 12 \
      --aws_ami {reddit | agentcompany}
   
   # For Qwen
   python run_qwen35vl_multienv.py \
      --observation_type screenshot \
      --agent_type Qwen35VLAgent \
      --headless \
      --model qwen3.5-plus-2026-02-15 \
      --coord relative \
      --enable_thinking \
      --result_dir ./results_qwen \
      --test_all_meta_path "./dataset/{subset}/{platform_instruction}/test_{k}_step.json" \
      --test_config_base_dir "./dataset/{subset}/{platform_instruction}"  \
      --max_steps {15*k} \
      --sleep_after_execution 4 \
      --provider_name aws \
      --num_envs 12 \
      --aws_ami {reddit | agentcompany}
   
   # For Claude
   python run_claude_multienv.py \
      --observation_type screenshot \
      --action_space pyautogui \
      --agent_type AnthropicAgent \
      --api_provider anthropic \
      --headless \
      --model {claude-sonnet-4-6 | claude-haiku-4-5} \
      --result_dir ./results_claude \
      --test_all_meta_path "./dataset/{subset}/{platform_instruction}/test_{k}_step.json" \
      --test_config_base_dir "./dataset/{subset}/{platform_instruction}"  \
      --max_steps {15*k} \
      --sleep_after_execution 4 \
      --provider_name aws \
      --num_envs 12 \
      --aws_ami {reddit | agentcompany}
   

   Arguments:

   • {subset}: Can be single_step (k=1), w_urgent (Urgent = Y), wo_urgent (Urgent = N).
   • {platform_instruction}: Can be forum, forum_follow, owncloud, owncloud_follow. They correspond to 2 platforms and 2 user instruction types.
   • {k}, {15*k}: The number of step number, $k \in {1,2,3}$. Valid (subset, k) combinations: single_step only with k=1; w_urgent and wo_urgent only with k \in \{2,3\}.
   • {reddit | agentcompany}: AWS AMI to launch. Should be reddit for forum platform or agentcompany for owncloud platform.

Automatic Decomposition Pipeline and Adding New Adversarial Goals

The code of Automatic Decomposition Pipeline is in auto_decomp/.

1. cd auto_decomp. Add new adversarial goals to be decomposed to adv_goals.py.
2. Run python iterative_refinement.py -k {k} for each k \in \{2, 3\}. Output: ir/goal_*_k{k}_N20_K3_all.json, containing the results of Stage 1.
3. Run python selected_examples/select_examples.py. Output: selected_examples/selected_examples.json that selects top-10 examples from Stage 1.
4. For Stage 2, write json test examples for CUA verification. Then, record the step number taken to select the final decomposition result.

Acknowledgement

StepJack builds on top of RedTeamCUA (Liao et al., 2025; arXiv:2505.21936), which itself is derived from OSWorld (XLANG NLP Lab, 2024). Both upstream sources are licensed under Apache-2.0.

Some StepJack data directories are not per-file headered because they contain binary or non-source fixtures. Their lineage is recorded here for attribution: