Learning to Detect Unknown Jailbreak Attacks in Large Vision-Language Models

Official implementation of “Learning to Detect Unknown Jailbreak Attacks in Large Vision-Language Models.”

This repository contains the data-processing, hidden-state extraction, classifier training, safety-pattern auto-encoder, and evaluation code used in the project.

This release provides the LLaVA-v1.6-Vicuna-7B implementation. Adapting the pipeline to Qwen2.5-VL or CogVLM requires model-specific input processing and separately trained detectors.

Contents

• Models
• Repository Structure
• Detection Pipeline
• Datasets

Models

The experiments use the following base models:

• LLaVA-v1.6-Vicuna-7B, a large vision-language model based on Vicuna.
• Llama Guard 3 8B, a safety guardrail model used to assess generated responses.

Download the model weights separately and place them under:

asset/weights/

Repository Structure

.
├── asset/
│   ├── advbench/                 # AdvBench images
│   ├── harmbench/                # HarmBench DirectRequest images
│   ├── GQA/                      # GQA images
│   ├── HiddenStates/             # Extracted hidden states
│   └── weights/                  # Model weights (not included)
├── Benchmarks/                   # Evaluation benchmark metadata
├── vicuna/
│   ├── instructions/
│   │   ├── advbench.json
│   │   ├── GQA.json
│   │   └── harmbench.json        # HarmBench DirectRequest metadata
│   ├── qa.py
│   ├── qa-baseline.py
│   └── train.py
├── autoencoder.py
├── llama3_guard.py
└── test.py

Detection Pipeline

Some scripts use paths relative to their own working directory. Run the commands from the directories shown below.

1. Query the vision-language model

Query the model on the unsafe source data (AdvBench) and safe source data (GQA):

cd vicuna
python qa.py --dataset advbench
python qa.py --dataset GQA
cd ..

The HarmBench DirectRequest data can be queried with the same interface:

cd vicuna
python qa.py --dataset harmbench
cd ..

2. Assess and split model responses

Assess the generated AdvBench responses with Llama Guard 3:

python llama3_guard.py --file vicuna/instructions/advbench.json

Create the AdvBench and GQA training/test splits:

cd vicuna/instructions
python process.py
cd ../..

3. Extract hidden states

Extract hidden states for the evaluation benchmarks currently configured in vicuna/qa-baseline.py:

cd vicuna
python qa-baseline.py
cd ..

4. Train and test the MSCAV classifiers

cd vicuna
python train.py --train
python train.py --test
cd ..

5. Train the Safety Pattern Auto-Encoder (SPAE)

python autoencoder.py

6. Evaluate detection performance

python test.py

Datasets

Training and appendix data

Dataset	Role	Metadata	Images
AdvBench	Unsafe source data	vicuna/instructions/advbench.json	asset/advbench/
GQA	Safe source data	vicuna/instructions/GQA.json	asset/GQA/
HarmBench (DirectRequest)	Unsafe-source experiment reported in the appendix	vicuna/instructions/harmbench.json	asset/harmbench/

The included HarmBench DirectRequest subset contains 320 image–request pairs. Image paths in harmbench.json are repository-relative and follow the same layout convention as AdvBench.

Evaluation benchmarks

Download the evaluation images from ModelScope: detecpolo/Learning-to-Detect and copy the downloaded asset/ directory into the repository root. The matching benchmark JSON files are provided in this repository under Benchmarks/.

Please refer to Section 4.1 and the appendices of our paper for dataset sources, attack construction, and experimental settings.