shuangliangx/learning-to-detectDetects unknown jailbreak attacks in large vision-language models using hidden state analysis and autoencoders, with training and evaluation pipelines for robust safety monitoring.
This repository provides the official implementation of "Learning to Detect Unknown Jailbreak Attacks in Large Vision-Language Models".
Our method uses the following two base models:
LLaVA-v1.6-Vicuna is a powerful vision-language model that combines a visual encoder with the Vicuna language model to process multimodal inputs and generate natural language responses.
LlamaGuard3 is a safety guardrail model developed by Meta AI, specifically designed to detect and prevent harmful content generation and effectively identify potentially unsafe requests and responses.
Please download the model weights and place them in the code/asset/weights directory.
(Optional, since the processed data and extracted states are already preserved in the repository.)
python code/vicuna/qa.py --file code/vicuna/instructions/advbench.json
python code/vicuna/qa.py --file code/vicuna/instructions/GQA.json
python code/llama3_guard.py --file code/vicuna/instructions/advbench.json
python code/vicuna/instructions/process.py
python code/vicuna/qa-baseline.py
python code/vicuna/train.py --train
python code/vicuna/train.py --test
python code/autoencoder.py
python code/test.py
| Dataset | Details |
|---|---|
| MM-SafetyBench | |
| HADES |
Please download the datasets and place them in the code/asset directory.