
Training-free, trigger-agnostic test-time detector for backdoored Vision Transformers that flags poisoned samples using inter-head attention consistency and entropy variance.
Siquan Huang1, Yijiang Li2, Xingfu Yan3,*, Ningzhi Gao1, Leyu Shi1, Ying Gao1,*
1South China University of Technology 2University of California, San Diego 3South China Normal University
*Corresponding authors
Vision Transformers (ViTs) are the visual encoder of most multimodal models, and practitioners routinely build on third-party pretrained checkpoints that may carry a backdoor. DetectViT is a test-time backdoor sample detector for ViTs that reads the attention heads instead of the model's outputs or embeddings.
On clean images, attention heads attend to different regions with comparable concentration. Backdoor triggers break this balance in two ways:
DetectViT extracts the CLS-to-patch attention distribution of every head and computes two complementary signals:
Thresholds are estimated with the three-sigma rule, $\tau = \mu + 3\sigma$, from only 64 to 128 clean out-of-distribution images of any source. A test sample is flagged as backdoored if $\mathcal{S}{\text{cons}} > \tau{\text{cons}}$ or $\mathcal{S}{\text{var}} > \tau{\text{var}}$.
Detection performance on four ViT backdoor attacks. TPR and FPR are in %, AUC is on a [0, 1] scale. "–" means the method is not applicable (label-based methods cannot handle free-form captions) or does not produce a continuous score.
| Defense | BadViT TPR↑ / FPR↓ / AUC↑ | TrojViT TPR↑ / FPR↓ / AUC↑ | BadCLIP TPR↑ / FPR↓ / AUC↑ | BadVision TPR↑ / FPR↓ / AUC↑ |
|---|---|---|---|---|
| TeCo | 77.95 / 61.86 / 0.63 | 55.00 / 51.76 / 0.51 | 99.20 / 39.52 / 0.89 | – |
| Scale-Up | 88.65 / 30.66 / 0.81 | 85.00 / 41.98 / 0.74 | 91.84 / 65.03 / 0.66 | – |
| PatchProc | 90.61 / 18.42 / 0.89 | 87.60 / 21.44 / 0.85 | 86.97 / 26.78 / 0.82 | – |
| DeDe | 63.97 / 23.29 / 0.78 | 50.20 / 25.80 / 0.72 | 55.12 / 27.97 / 0.69 | 7.00 / 29.68 / 0.50 |
| BackdoorIDS | 80.79 / 9.73 / – | 5.00 / 10.84 / – | 93.24 / 16.88 / – | 96.00 / 15.96 / – |
| DetectViT | 100.00 / 1.58 / 1.00 | 90.80 / 5.94 / 0.95 | 100.00 / 0.17 / 1.00 | 96.00 / 3.00 / 0.99 |
Additional attacks. On five further attacks covering self-supervised encoders (BadEncoder, DRUPE), clean-label poisoning (Label-Consistent), frequency-domain triggers (Low-Frequency) and optimized global triggers (Blended-Optimized), DetectViT is the only method that keeps an AUC of at least 0.97 on all of them. Full results are in the appendix of the paper.
Runtime. Average per-sample detection time (s):
| Defense | BadViT | TrojViT | BadCLIP | BadVision |
|---|---|---|---|---|
| TeCo | 2.5094 | 4.2920 | 1.8448 | – |
| Scale-Up | 0.0246 | 0.0261 | 0.0696 | – |
| PatchProc | 0.6686 | 0.6926 | 0.8257 | – |
| DeDe | 0.0206 | 0.0279 | 0.1034 | 0.6060 |
| BackdoorIDS | 0.1734 | 0.1982 | 0.2560 | 3.3458 |
| DetectViT | 0.0124 | 0.0134 | 0.0163 | 0.0226 |
Click the image to open the full-resolution PDF.
git clone https://github.com/siquanhuang/DetectViT.git
cd DetectViT
Download the pretrained backdoored model checkpoints and optimized triggers from Google Drive:
The detection method is implemented in detectvit.py. Run main.py to evaluate DetectViT against each attack.
python3 main.py \
--defense DetectViT \
--attack_type badvit \
--trojvit_attack_dir TrojViT/logs/Caltech101_DeiT-T_target0_BadViT \
--trojvit_network DeiT-T \
--trojvit_num_classes 102 \
--trojvit_data_dir ./data
python3 main.py \
--defense DetectViT \
--attack_type trojvit \
--trojvit_attack_dir TrojViT/logs/Tiny-ImageNet_DeiT-S_target2_TrojViT \
--trojvit_data_dir ./data/tiny-imagenet-200 \
--trojvit_network DeiT-S \
--gpu 0
python3 -u main.py \
--defense DetectViT \
--attack_type badclip \
--gpu 0 \
--eval_data_type ImageNet1K \
--eval_test_data_dir BadCLIP/data/ImageNet1K/validation/ \
--add_backdoor \
--asr \
--label banana \
--patch_type ours_tnature \
--patch_location middle \
--patch_name BadCLIP/opti_patches/tnature_eda_bs64_ep50_32_middle_01_05_pos_neg.jpg \
--patch_size 32
python3 -u main.py \
--defense DetectViT \
--attack_type badvision \
--gpu 0
| Argument | Description | Default |
|---|---|---|
--defense | Detection method (DetectViT, DeDe, BackdoorIDS, ...) | DetectViT |
--attack_type | Attack to evaluate (badvit, trojvit, badclip, badvision) | — |
--gpu | GPU device ID | 0 |
If you find this work useful, please consider citing our paper.