soildworks/agentic-clip-benchmarkAn automated, high-precision zero-shot evaluation pipeline for OpenAI's CLIP model on CIFAR-10. Features 88.80% accuracy, Safetensors security mitigation (CVE-2025-32434), and AI Native (Trae) workflow.
-green.svg){kind=icon}
This project implements a robust, automated pipeline for evaluating OpenAI's CLIP (ViT-B/32) model on the full CIFAR-10 test set (10,000 images). Developed using an AI Native workflow (Trae IDE), it achieves a high-precision zero-shot accuracy of 88.80%.
88.80% (Zero-Shot)openai/clip-vit-base-patch32 (using Safetensors)
Analysis: The model shows strong diagonal dominance. Statistical bias is observed between 'Deer' and 'Horse' (15.4% overlap), likely due to similar skeletal contours in the ultra-low 32x32 resolution of CIFAR-10. Automobile and Horse categories achieved the highest precision (97.6%).
CIFAR10Dataset to handle raw binary pickle data, converting 1D byte-arrays into 3D RGB tensors efficiently.torch.utils.data.DataLoader with tqdm to manage 10,000 samples without VRAM overflow."a photo of a {label}" to optimize the semantic alignment between text and image embeddings.scikit-learn to compute the confusion matrix, providing a granular look at semantic classification errors.This project documents the resolution of several production-level challenges encountered during development:
.bin (Pickle) files.use_safetensors=True for zero-copy, secure model loading.HF_HUB_DISABLE_SYMLINKS=1 and established a manual cache cleanup protocol to ensure cross-platform compatibility.CLIPProcessor pipeline to handle raw uint8 inputs correctly, successfully restoring accuracy to 88.8%.HF_ENDPOINT mirror and local_files_only=True for high-speed, offline-capable deployment.pip install torch transformers pillow numpy tqdm scikit-learn seaborn matplotlib
Download the CIFAR-10 Python version and place the cifar-10-batches-py folder in the project root.
# On Windows, simply run the pre-configured batch file:
.\run.bat
# Or manually via Python:
python full_cifar_clip_eval.py
clip_eval_project/
├── full_cifar_clip_eval.py # Core evaluation script
├── run.bat # One-click run entry
├── confusion_matrix.png # Confusion matrix result
├── cifar-10-batches-py/ # CIFAR-10 raw data
│ └── test_batch # Test set data
└── README.md # Project documentation
This project is licensed under the MIT License. See the LICENSE file for details.