Skip to content
KitploitKITPLOIT
ToolsExploitsBlog
Log in
Submit
ToolsExploitsBlog
Submit

Hacking, PenTest, and Cybersecurity Tools for Your Security Arsenal!

Kitploit is a directory of hacking, cybersecurity, and pentesting tools. Discover the latest project updates to find vulnerabilities, analyze systems, automate testing, and strengthen your security.

FeedsContactPrivacy© 2026 Kitploit

Tool Directory

Categories

View all categories
Loading categories
Tools/GitHubGitHub/borealisai/stepjack
Dynamic Analysis (Sandboxing)Security VirtualizationRed TeamingAI SecurityAdversarial Attack
GitHubborealisai/stepjack

StepJack

Benchmarking framework for evaluating computer-use AI agents against multi-step indirect prompt injection, with automatic adversarial goal decomposition and AWS-sandboxed test environments.

View Repository
72476 months agoNot yet reviewed
Share

StepJack: Benchmarking Computer-Use Agent Safety Against Multi-Step Indirect Prompt Injection

Setup

1. Python Environment Setup

First, install all dependencies and ensure you’re running Python 3.10.

conda create -n stepjack python=3.10
conda activate stepjack

pip install -r requirements.txt

2. Sandbox Environment Setup

We use the AWS environment provided by RedTeamCUA.

AWS

  1. Configure NETWORK_INTERFACE_MAP:

    Configure the NETWORK_INTERFACE_MAP in aws_config.py based on your own AWS instances.

  2. Configure your security group:

    It’s important to configure a security group that allows legitimate access to your hosted instances while blocking unauthorized access attempts. This is crucial because the AMI, by default, uses only basic password-based authentication, which is relatively easy to compromise.

    Your local machine will access the services hosted on AWS via HTTP/TCP connections. Therefore, you should explicitly allow access to the following ports: 8092, 6379, 9999, 5000, 9222, 8091, 2999, and 3000, which are used by either WebArena or TheAgentCompany.

    In addition, we use SSH to directly modify the underlying database in order to simulate injection scenarios. As such, SSH access from your local IP address must also be allowed in the security group settings.

  3. Configure your AWS credentials by using aws configure, which will prompt you for:

    • AWS Access Key ID
    • AWS Secret Access Key
    • Default region name (Optional, you can press enter)

Experiment

scripts/run_gpt54_2step.sh shows one example of running GPT-5.4 CUA on StepJack.

  1. Set Environment Variables

    CUA API keys

    # For GPT 
    unset OPENAI_BASE_URL
    export OPENAI_API_KEY="sk-proj-XXXX"
    
    # For Kimi
    export KIMI_API_KEY='sk-XXXX'
    
    # For Qwen
    export OPENAI_BASE_URL="https://dashscope-us.aliyuncs.com/compatible-mode/v1"
    export OPENAI_API_KEY="sk-XXXX"
    
    # For Claude
    export ANTHROPIC_API_KEY='sk-XXXX'
    

    AWS Credentials

    export AWS_REGION='XX'
    export AWS_ACCESS_KEY='XX'
    export AWS_SECRET_KEY='XX'
    
  2. Run the Experiments

    # For GPT
    python run_gpt54_multienv.py \
       --observation_type screenshot \
       --agent_type GPT54Agent \
       --headless \
       --model gpt-5.4-mini-2026-03-17 \
       --result_dir ./results_gpt54 \
       --test_all_meta_path "./dataset/{subset}/{platform_instruction}/test_{k}_step.json" \
       --test_config_base_dir "./dataset/{subset}/{platform_instruction}"  \
       --max_steps {15*k} \
       --sleep_after_execution 4 \
       --provider_name aws \
       --num_envs 12 \
       --aws_ami {reddit | agentcompany}
    
    # For Kimi
    python run_kimi_multienv.py \
       --observation_type screenshot \
       --agent_type KimiAgent \
       --headless \
       --model Kimi-K2.5 \
       --result_dir ./results_kimi \
       --test_all_meta_path "./dataset/{subset}/{platform_instruction}/test_{k}_step.json" \
       --test_config_base_dir "./dataset/{subset}/{platform_instruction}"  \
       --max_steps {15*k} \
       --sleep_after_execution 4 \
       --provider_name aws \
       --num_envs 12 \
       --aws_ami {reddit | agentcompany}
    
    # For Qwen
    python run_qwen35vl_multienv.py \
       --observation_type screenshot \
       --agent_type Qwen35VLAgent \
       --headless \
       --model qwen3.5-plus-2026-02-15 \
       --coord relative \
       --enable_thinking \
       --result_dir ./results_qwen \
       --test_all_meta_path "./dataset/{subset}/{platform_instruction}/test_{k}_step.json" \
       --test_config_base_dir "./dataset/{subset}/{platform_instruction}"  \
       --max_steps {15*k} \
       --sleep_after_execution 4 \
       --provider_name aws \
       --num_envs 12 \
       --aws_ami {reddit | agentcompany}
    
    # For Claude
    python run_claude_multienv.py \
       --observation_type screenshot \
       --action_space pyautogui \
       --agent_type AnthropicAgent \
       --api_provider anthropic \
       --headless \
       --model {claude-sonnet-4-6 | claude-haiku-4-5} \
       --result_dir ./results_claude \
       --test_all_meta_path "./dataset/{subset}/{platform_instruction}/test_{k}_step.json" \
       --test_config_base_dir "./dataset/{subset}/{platform_instruction}"  \
       --max_steps {15*k} \
       --sleep_after_execution 4 \
       --provider_name aws \
       --num_envs 12 \
       --aws_ami {reddit | agentcompany}
    

    Arguments:

    • {subset}: Can be single_step (k=1), w_urgent (Urgent = Y), wo_urgent (Urgent = N).
    • {platform_instruction}: Can be forum, forum_follow, owncloud, owncloud_follow. They correspond to 2 platforms and 2 user instruction types.
    • {k}, {15*k}: The number of step number, $k \in {1,2,3}$. Valid (subset, k) combinations: single_step only with k=1; w_urgent and wo_urgent only with k \in \{2,3\}.
    • {reddit | agentcompany}: AWS AMI to launch. Should be reddit for forum platform or agentcompany for owncloud platform.

Automatic Decomposition Pipeline and Adding New Adversarial Goals

The code of Automatic Decomposition Pipeline is in auto_decomp/.

  1. cd auto_decomp. Add new adversarial goals to be decomposed to adv_goals.py.
  2. Run python iterative_refinement.py -k {k} for each k \in \{2, 3\}. Output: ir/goal_*_k{k}_N20_K3_all.json, containing the results of Stage 1.
  3. Run python selected_examples/select_examples.py. Output: selected_examples/selected_examples.json that selects top-10 examples from Stage 1.
  4. For Stage 2, write json test examples for CUA verification. Then, record the step number taken to select the final decomposition result.

Acknowledgement

StepJack builds on top of RedTeamCUA (Liao et al., 2025; arXiv:2505.21936), which itself is derived from OSWorld (XLANG NLP Lab, 2024). Both upstream sources are licensed under Apache-2.0.

Some StepJack data directories are not per-file headered because they contain binary or non-source fixtures. Their lineage is recorded here for attribution:

Download Tool

Most Popular

View all →

Discover the most used tools by our community.

  • 1

    OsintHunter

    letox4/osinthunter

    🎯 The ultimate OSINT toolkit: investigation graph, confidence scoring, alias families, map, timeline and reports. 49 modules, 400+ sites, no API keys required.

    772
  • 2

    ghostlock-app

    yukonga/ghostlock-app

    GhostLock One-Tap Execution App (CVE-2026-43499)

    505
  • 3

    ARTEX

    baarai/artex

    LLM multi-agent autonomous penetration testing platform with planner/worker agents, asset and exploration graphs, MITM traffic recording, and vulnerability retesting.

    420
  • 4

    redforge

    cybersecurity91/redforge

    Browser-side red team toolkit with 102 offline tools for recon, payload generation, credential attacks, crypto, encoding and engagement reporting.

    345
  • 5

    CyberStrikeAI

    ed1s0nz/cyberstrikeai

    The system of action for AI-native cybersecurity—where intent becomes governed execution, evidence becomes operational memory, and every operation improves the next.

    260
  • 6

    antifox

    antifox-ai/antifox

    AI multi-agent autonomous penetration testing platform that runs recon, attack-surface discovery, exploitation validation and evidence-backed report delivery.

    251

Explore all tools

Browse our collection of tools

View all tools →