Telling identical GPUs apart

Blog post: Telling identical GPUs apart from their numerical output

Two GPUs of the same model compute the same sum in a slightly different order, and that difference is enough to tell them apart. A small CUDA kernel identifies which of 19 identical NVIDIA A40s it ran on with 99.1% accuracy, using only the numbers it computes. It reads no serial number and uses no timers.

• It survives reboots. After a container restart, all 12 measurements matched the right chip.
• It couldn't be forged. An attacker on one chip, tuning every launch setting it controls, reached at best 0.616 correlation with another chip's fingerprint. Genuine chips match their own at 0.946 on average, and never below 0.916.
• A default torch.index_add_ call leaks it too. Reading nothing but that op's ordinary output identifies the chip with 100% accuracy among GPUs on the same driver, and 78% across mixed drivers.
• It works across models: A40, RTX 3090, RTX 4090 and L40S.

Why it works: when thousands of thread blocks race to update one counter, the hardware decides who arrives first. Tiny manufacturing differences between a chip's cores bias that order the same way every time.

PyTorch's deterministic mode removes the index_add_ leak. The repo contains the probes, every measurement behind the numbers below, and scripts that reproduce each result on a laptop CPU.

Results

Result	Number	Reproduce
Active probe, 19 identical A40s, P3 channel (numerical output only, no %smid)	99.1% LOO (111/112), chance 5.3%, separation 4.28x	analysis/pooled_analyze.py
Active probe, 8 A40s	P3 100% LOO, separation 4.09x, permutation p = 0.0005	analysis/analyze_puf.py
Driver confound: 4 A40s on the identical driver (580.159.04)	P3 100% (6.10x), P2 100% (5.43x)	analysis/analyze_puf.py
Reboot: container restart, same die verified by UUID (4 A40s)	100% re-identification (12/12), both channels	analysis/temporal_analyze.py
RTX 3090, 7 chips across 4 driver versions	P3 100% LOO (4.55x), P2 100% (8.12x), p = 0.0005	analysis/cross_model_analyze.py
RTX 4090, 2 chips on the same host and driver	P3 6/6 (12.5x), P2 6/6 (17.8x)	analysis/cross_model_analyze.py
L40S, 2 chips	P3 4/4 (17.2x), P2 4/4 (13.8x)	analysis/cross_model_analyze.py
Passive torch.index_add_, K=65536, 6 A40s	100% same-driver (9/9), 78% cross-driver	analysis/passive_analyze.py
Passive signature capacity, K=65536	~217 independent bits (A40), ~239 (RTX 4090)	analysis/roc_capacity.py
Forgery: attacker sweeps every launch knob it controls	best forge 0.616 correlation vs 0.92 genuine floor	analysis/forge_analyze.py
Boundary: vanilla transformer fwd/bwd, dense GEMM, TF32 matmul	bit-identical across chips, no leak	analysis/leak_analyze.py, analysis/channel_analyze.py
Defense: torch.use_deterministic_algorithms(True)	passive channel eliminated; 2.5-3.6x backward cost; active probe unaffected	analysis/defense_analyze.py

LOO = leave-one-out nearest-centroid classification of single measurements against per-chip centroids. Separation = between-chip distance / within-chip distance.

How it works

The probe (probes/probe_puf.py) is a small cupy kernel. It launches 4096 blocks of 128 threads. Every thread does the same fixed chain of FMAs, then thread 0 of each block takes a ticket from a global counter with atomicAdd(counter, 1). The ticket is that block's arrival slot. Nothing about the input differs between chips; the only thing that varies is which blocks get there first.

That order is not random. The work distributor maps block indices onto SMs in a near-fixed pattern, and each SM on a given die runs slightly faster or slower than its siblings because of manufacturing variation. Averaged over 3000 launches, each block index settles to a mean arrival rank that reflects the speed of the SM it lands on.

fixed kernel, fixed input
        |
        v
blocks race to atomicAdd  -->  arrival slot per block (hardware-decided)
        |
        v  average over 3000 launches
mean arrival rank per block index (P3)  /  per physical SM via %smid (P2)
        |
        v  subtract the cross-chip mean (removes the architecture)
per-chip residual  -->  nearest-centroid match against enrolled chips

Three channels come out of one run:

• P3, per-block rank. Mean arrival slot per block index. It is read from the kernel's numerical output alone: no %smid, no timers, no privileged counters. This is the headline channel.
• P2, per-SM rank. The same slots binned by physical SM via the %smid register. Binning by physical SM averages out thermal noise, so P2 holds up under load: 100% re-identification (18/18) after 60 s of full-load stress.
• P1, marginal sum. A histogram of the float32 result of a fixed reduction. It is weak (1.20x separation on 8 A40s), because summing destroys most of the order information.

Subtracting the mean across chips removes everything the chips share (architecture, driver, compiler) and leaves the per-die residual. Two measurements of the same chip land close together; measurements of different chips do not.

Quickstart

git clone https://github.com/divagr18/gpu-instance-fingerprint.git
cd gpu-instance-fingerprint
pip install -r requirements.txt        # numpy + scipy, enough for every analysis

Run the analysis on the included data (CPU, no GPU needed):

python analysis/pooled_analyze.py      # 19 A40s: 111/112 = 99.1%
python analysis/analyze_puf.py "a40_8chip/*.json"   # 8 A40s: P3 100%

Run the probe on a GPU:

pip install numpy cupy-cuda12x
python probes/probe_puf.py 3 0 > gpu_a_r0.json   # 3 epochs, replicate 0 (about 2 s per epoch)
python probes/probe_puf.py 3 1 > gpu_a_r1.json

Collect a few replicates from each of two or more GPUs of the same model, then:

python analysis/analyze_puf.py "path/to/*.json"

Each JSON records the GPU UUID, driver, cupy version and per-epoch clock and temperature next to the measurements. The analysis groups by model name and labels chips by UUID.

To run on remote machines over SSH, scripts/run_remote.sh pushes a probe, runs it on several hosts in parallel and collects the JSON:

OUT_DIR=data/my_run REPS=4 scripts/run_remote.sh probe gpu1=root@HOST1:PORT gpu2=root@HOST2:PORT

docker/Dockerfile builds an image with CUDA 12.8, cupy and the probe preinstalled (docker build -f docker/Dockerfile -t gpu-instance-fingerprint .).

Reproducing each result

All commands run from the repo root on a CPU. Scripts find their data in data/ on their own.