
A black-box optimization approach for protecting Large Language Model (LLM) system prompts against adversarial extraction attacks.
A black-box optimization approach for protecting Large Language Model (LLM) system prompts against adversarial extraction attacks.
This repository implements Prompt Sensitivity Minimization (PSM), a defense mechanism that uses LLM-as-optimizer to automatically generate protective "shields" around system prompts. PSM minimizes observable leakage while preserving task utility through an iterative optimization process.
git clone <repository-url>
cd psm
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
pip install -r requirements.txt
config/config.yaml file with your LLM API configurations:llms:
gpt-4o-mini:
model: "gpt-4o-mini"
model_provider: "openai"
api_key: "your-api-key"
temperature: 0.7
# Add other model configurations as needed
Generate protected system prompts using PSM:
python run.py
This will:
data/victim_prompts/data/defense_prompts/Evaluate defense mechanisms against adversarial attacks:
python experiments/evaluate_defenses.py
This will:
results/from run import PSMRunner, RunPSMConfig
# Configure PSM
config = RunPSMConfig(
dataset_name="unnatural-test",
attack_samples=50,
validation_samples=10,
target_dataset_samples=30,
llm_target="gpt-5-mini",
llm_optimizer="gpt-4o-mini",
llm_validation="gpt-4o",
llm_judge="gpt-4o-mini",
n_optimization_iterations=10,
n_initial_shields=5,
n_shields_per_step=5,
max_candidates_per_step=10,
target_utility_threshold=0.9,
target_leakage_threshold=0.65,
leakage_aggregation="logsumexp",
logsumexp_temperature=10.0
)
# Run PSM
runner = PSMRunner(config)
runner.run_experiments()
from experiments.evaluate_defenses import (
DefenseEvaluator, ExperimentConfig,
DefenseConfig, DefenseStrategy,
AttackType, Dataset, TargetModel
)
# Configure experiment
config = ExperimentConfig(
attack_types=[AttackType.RACCON, AttackType.LIANG],
defense_configs=[
DefenseConfig(DefenseStrategy.PSM, Dataset.UNNATURAL_TEST, TargetModel.GPT_4O_MINI)
],
max_victim_samples=30,
max_attack_samples=60,
use_am_metric=True,
am_threshold=0.9,
use_jm_metric=True,
jm_threshold=0.7
)
# Run evaluation
evaluator = DefenseEvaluator(config)
await evaluator.run_experiments()
evaluator.save_results()
evaluator.print_results_summary()
src/defense/psm/psm.py)The main optimization engine that:
Key Classes:
PSM: Main optimization classPromptCandidate: Represents a candidate solution with scoresPSMConfig: Configuration parameterssrc/defense/psm/psm_data_creation.py)Handles:
src/metrics.py)Evaluation metrics:
experiments/evaluate_defenses.py)Comprehensive evaluation framework:
Supported attack datasets:
raccon.json: RACCON attack promptsliang.json: Liang et al. attack patternszhang.json: Zhang et al. attack patternsours.json: Custom attack promptsraccon_language.json: Language-specific RACCON attacks@dataclass
class RunPSMConfig:
# Dataset
dataset_name: str = "unnatural-test"
# Sample sizes
attack_samples: int = 50 # Attack inputs to use
validation_samples: int = 10 # Baseline validation queries
target_dataset_samples: int = 30 # System prompts to process
# Model selection
llm_target: str = "gpt-5-mini" # Target model to protect
llm_optimizer: str = "gpt-4o-mini" # Model for shield generation
llm_validation: str = "gpt-4o" # Model for validation
llm_judge: str = "gpt-4o-mini" # Model for judging
# Optimization parameters
n_optimization_iterations: int = 10 # Max optimization iterations
n_initial_shields: int = 5 # Initial shield population
n_shields_per_step: int = 5 # New shields per iteration
max_candidates_per_step: int = 10 # Max candidates to consider
# Thresholds
target_utility_threshold: float = 0.9 # Minimum utility score
target_leakage_threshold: float = 0.65 # Maximum leakage score
utility_penalty_weight: float = 100.0 # Weight for utility penalty
# Leakage aggregation
leakage_aggregation: str = "logsumexp" # Aggregation method
logsumexp_temperature: float = 10.0 # LogSumExp temperature
top_k: int = 3 # Top-k for top_k_mean
config = ExperimentConfig(
# Attack and defense selection
attack_types=[AttackType.RACCON, AttackType.LIANG],
defense_configs=[...],
# Sample limits
max_victim_samples=30,
max_attack_samples=60,
# AM metric
use_am_metric=True,
am_threshold=0.9,
# JM metric
use_jm_metric=True,
jm_threshold=0.7,
# Output options
output_dir="results",
save_detailed_results=True,
save_per_prompt_results=True
)
This section provides detailed instructions for reproducing the experimental results from this repository.