
Is 4-bit quantization more than noise to a chat model's safety behaviour? Code, pre-registration and labels.
Attributing behavioural shifts with matched perturbations
Code and analysis for a pre-registered audit of how 4-bit quantization changes the refusals of small chat models, with controls that ask whether a change comes from quantization or would follow from any perturbation of the same size.
Paper: arXiv:2611.22186
Chat models that run on phones and laptops are usually 4-bit copies of the model that was evaluated for safety. Audits of such copies disagree: some find that quantization weakens refusal, some find it does not, some find that models refuse more. A 4-bit build is one particular perturbation of every weight, and there are many other perturbations of the same size. So a shift in its refusals could come from the structure of rounding error, or from the model's own lean under any nudge of that size.
We test three 3-4B chat models (Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct, Qwen3-4B-Instruct-2507) in the round-to-nearest formats that MLX ships for Apple Silicon. Every comparison is paired and read against an 8-bit build, which leaves likelihood unchanged. In brief:
The numbers behind every sentence in the paper are generated by one script (see Reproducing the paper).
| Path | Contents |
|---|---|
PREREG.md | The pre-registration and its five dated addenda, each fixed before the data it concerns |
qrefusal/ | Shared code: chat formatting with a pinned template date, greedy and sampled decoding, a layer-by-layer forward pass with hooks (MLX has no hook API), refusal directions, paired statistics |
runs/ | One script per experiment; each writes to results/<experiment>/ |
analysis/make_numbers.py | Regenerates every number in the paper from the raw result files, with integrity checks |
analysis/figures.py | Draws the figures from paper/numbers.json |
paper/ | LaTeX source; numbers enter the text only as macros generated by the analysis |
scripts/ | Model conversion, data fetching, and the run queues |
tests/ | Identity tests for the hooked forward pass and the statistics |
| Paper | Script | What it does |
|---|---|---|
| §3 Setup | scripts/convert.sh | Builds bf16, 8-bit, 4-bit (group 64 and 32), mixed 4/6, MXFP4, AWQ and GPTQ copies of a model |
| §3 Setup | runs/r2.py, runs/e7_abliterate.py, runs/e7b_prefixes.py | Refusal directions per layer; an abliterated bf16 copy; the compliant prefixes used for prefill |
| §3 Setup | runs/r1b_judge.py, runs/j2_ensemble.py, runs/l2_pair_sheet.py | Content judge (Qwen2.5-14B), Llama Guard 3, and the blind two-phase human-labelling sheet |
| §4 Label instability and the 8-bit null | runs/e0_determinism.py, runs/a2_depth.py --temp | Greedy rerun identity; the sampling seed floor |
| §5.1 Forced start | runs/a2_depth.py, runs/r1.py | Prefill depths k = 0, 2, 8; generic prefix (--fixed_prefix); length check (--max_tokens 384 --sample 150) |
| §5.2 Free start | runs/x2_sneaky.py | WildJailbreak development and held-out test sets; SORRY-Bench |
| §6 Two simpler explanations | runs/x1_xstest.py, runs/c1_capability.py | Over-refusal on XSTest and jailbreak-styled benign prompts; likelihood and bits per weight |
| §7 Matched-perturbation controls | runs/e2_perturb.py, runs/e2_errors.py, runs/g1_gguf.py | Matched-perturbation and scaled builds with a validation gate; per-matrix rounding error; the llama.cpp runtime |
| §8 Refusal-direction projections | runs/m1_direction.py | Projections onto the bf16 refusal direction |
| Appendix | runs/a1_prefill.py, runs/r5_runon.py | The prefill pilot; an end-of-turn failure in MXFP4 Qwen2.5 |
Hardware: one Apple Silicon Mac with 16 GB of memory (we used an M2 Pro). The full set of runs takes several days of machine time; models are loaded one at a time.
pip install -e ".[test]" # mlx 0.32.4, mlx-lm 0.32.2
scripts/fetch_data.sh # refusal_direction prompt splits (Apache-2.0)
scripts/convert.sh Qwen/Qwen2.5-3B-Instruct qwen2.5-3b
scripts/convert.sh meta-llama/Llama-3.2-3B-Instruct llama3.2-3b
scripts/convert.sh Qwen/Qwen3-4B-Instruct-2507 qwen3-4b
scripts/convert.sh Qwen/Qwen2.5-0.5B-Instruct qwen2.5-0.5b # needed by the tests and Appendix G
python -m pytest tests
Three prompt sets are obtained from their authors and placed under data/:
data/xstest/xstest_prompts.csvdata/sorry/question*.jsonldata/wildjailbreak/eval/eval.tsvThe run queues in scripts/ give the commands for the later registered analyses in the order we ran them; the table above gives the scripts for the rest. They expect a conda environment named ml, and runs/g1_gguf.py expects llama.cpp (commit 0ec4027) at ../tools/llama.cpp-src and Homebrew llama-quantize. Each experiment skips outputs that already exist, so a queue can be stopped and restarted. Then:
python analysis/make_numbers.py # paper/numbers.{md,json}, paper/tex/numbers.tex
python analysis/figures.py # paper/tex/figures/*.pdf
cd paper/tex && tectonic main.tex
PREREG.md fixes the primary test, the secondary families, the human-label design, the held-out free-start test and the controls' decision rule. Each addendum is dated and was fixed before the data it concerns existed. The paper's appendix lists what was registered when and the changes to the plan. The dated commits are in a private development repository (hashes 50d0225, 2c53665, 86c93e0, 3e2ea52, 02e72c5, 09e49aa), available on request; this repository's history starts at release. Three sentences that pointed to private planning notes were removed from PREREG.md; nothing else was changed.
Released: the code, the pre-registration, the aggregate results (paper/numbers.{md,json}) and every judge, Llama Guard and human label, with no prompt or reply text, in labels/ (written by analysis/export_labels.py). Rows are keyed by setting, model, build and prompt index i: the row index of the prompt split that runs/ loads (for x2/*, the seeded samples in runs/x2_sneaky.py). Not released: the model replies, which include harmful content produced under a forced start; make_numbers.py reads them from results/, which is excluded from version control. They are available to researchers on request.
Code: MIT (LICENSE). Labels and paper text: CC BY 4.0. paper/tex/acl.sty and acl_natbib.bst keep their own terms.
@article{kaya2026quantization,
title={Is 4-Bit Quantization More Than Noise to a Chat Model's Safety Behaviour? Attributing Behavioural Shifts with Matched Perturbations},
author={Kaya, Ege},
journal={arXiv preprint arXiv:2611.22186},
year={2026}
}