LM-Fingerprint

Fingerprints the serving stack behind an OpenAI-compatible chat endpoint.

It records tokenizer accounting, hidden template offset, validation prose, JSON/SSE shape, role acceptance, named limits, and gateway headers. It does not load weights, store API keys, or ask the model who it is.

Authorized use only: endpoints you own, labs you control, in-scope bounties, written pentests. See SECURITY.md. --api-key-env names an environment variable. The key is never logged.

Contents

1. Problem
2. What a fingerprint is
3. Architecture
4. Threat model
5. Install and the repro gate
6. First run
7. Against a live endpoint
8. Local Hugging Face tokenizers
9. CLI
10. Library
11. Reading a result JSON
12. Measured vs ModelPrint
13. Honesty rules
14. Nearby tools
15. License

Problem

OpenAI-compatible chat is a common wire. vLLM, SGLang, a lab gateway, and a reseller can all speak /chat/completions. The model field is a label. The host can swap the backend, wrap the tokenizer, or sit a router in front. Personality probes move when the system prompt moves.

What stays put is plumbing: how the server counts tokens, how many hidden template tokens it adds, the prose its validator wrote, the SSE dialect, which roles it accepts, the number it names when max_tokens is absurd, the Server / provider headers.

That is the question this tool answers: what stack is handling this chat path? Not which checkpoint, not which process is bound to the port, not who wrote a completion.

ModelPrint asked the same question in a browser with nine probes. This package is the CLI and library form: versioned probes, enrollable library, pairwise evidence, drift, and a local tokenizer path that never calls a hosted Inference API.

What a fingerprint is

A fingerprint is a versioned JSON record of probe values. Each probe must return the same value when the same stack is hit twice. Timestamps, request ids, latency, and keys are stripped.

Tokenizer counts reuse ModelPrint 1.0.0 texts (MIT, unclecode/modelprint), byte-exact, minus a one-character "a" baseline so a hidden template cancels. Those four numbers are a projection of the encode map (T). Equal counts do not mean (T_1 \equiv T_2). GPT-2 and OPT collide on them; the rust pipeline hash does not.

hf_identity (local HF path only) is SHA-256 of that pipeline commitment plus the chat-template hash. Same (T) and the same template match (tiny-gpt2 / gpt2). That is family + template, not a checkpoint.

Architecture

Layout:

src/lm_fingerprint/
  cli.py         argparse. --api-key-env only. No --api-key.
  client.py      HttpTransport, InProcessTransport, redact_secrets
  probes.py      23 ProbeSpec rows (tokenizer / errors / shape / roles / limits / proxy)
  engine.py      run suite → Fingerprint
  schema.py      schema_version=1, probe_suite_version=lm-fingerprint-probes-v1
  similarity.py  weighted exact-match, coverage, confidence, library rank, drift
  library.py     enroll / match. Packaged rows are synthetic fixtures
  synthetic.py   in-process stacks for tests and the shipped library
  hf.py          local AutoTokenizer + named validation profile. No weights
  localmap.py    encode-pipeline-v2 commitment (rust tokenizer.json)
  proof.py       measured ModelPrint 9-probe identity
  report.py      text reports
  data/library.json

Two paths into the same probe suite:

live endpoint                         local Hub tokenizer (optional [hf])
     |                                         |
 HttpTransport.chat                      HfChatTransport
 POST /chat/completions                  encode / chat_template for counts
     |                                   synthetic profile for errors/shape
     +------------------+----------------------+
                        |
                   engine.run_probes
                        |
              Fingerprint (JSON, keyless)
                        |
          compare | match | drift | enroll

Probe authoring: docs/PROBES.md. Module map: docs/ARCHITECTURE.md.

Threat model

A match is shared infrastructure. One lab can serve two checkpoints on one stack. A router can answer with its own error wrapper. Count agreement is not identity of (T).

Install and the repro gate

Python 3.10+. Core runtime is the standard library.

git clone https://gitlab.com/WattoCyber/lm-fingerprint.git
cd lm-fingerprint
pip install -e ".[dev]"
PYTHONPATH=src python3 scripts/repro.py

Expect REPRO_OK. That gate is offline: no GPU, no production APIs. Local HF tests skip if the tokenizer is not cached.

pip install -e ".[hf]"    # transformers, for fingerprint-hf / verify-hf

First run

lm-fingerprint list-probes
lm-fingerprint list

list prints the packaged stacks: openai-chat-strict, glm-compat, router-openai, permissive-compat. Those are fixtures, not scraped providers. Compare two of them with no network:

from lm_fingerprint.client import InProcessTransport
from lm_fingerprint.engine import fingerprint_endpoint
from lm_fingerprint.similarity import compare_fingerprints
from lm_fingerprint.synthetic import SYNTHETIC_STACKS, handler_for

def fp(stack_id):
    t = InProcessTransport(handler_for(stack_id), headers=SYNTHETIC_STACKS[stack_id].headers)
    return fingerprint_endpoint(t, model=f"synthetic/{stack_id}", base_url=f"inprocess://{stack_id}")

print(compare_fingerprints(fp("openai-chat-strict"), fp("router-openai"))["note"])

Same word tokenizer, different errors and headers. Score lands around 0.58. ModelPrint's four counts match; this tool does not call that a shared stack.

Against a live endpoint

lm-fingerprint fingerprint \
  --base-url https://your-lab.example/v1 \
  --model the-label \
  --api-key-env OPENAI_API_KEY \
  --out lab.json

lm-fingerprint match --fingerprint lab.json
lm-fingerprint enroll --fingerprint lab.json --stack-id my-lab --library my-lib.json

Do not pass the key on the command line. Do not scrape production APIs to fill a public library.

Local Hugging Face tokenizers

lm-fingerprint fingerprint-hf --model sshleifer/tiny-gpt2 --local-files-only
lm-fingerprint verify-hf --local-files-only

Counts come from apply_chat_template when a template exists, else encode(..., add_special_tokens=False). Error taxonomy comes from a named synthetic profile (default openai-chat-strict) so the run stays offline. Remote handles (https://, api://, …) are rejected.

CLI

lm-fingerprint list-probes
lm-fingerprint list [--library PATH]
lm-fingerprint fingerprint --base-url URL --model MODEL --api-key-env VAR [--out PATH]
lm-fingerprint fingerprint-hf --model HANDLE [--profile NAME] [--local-files-only]
lm-fingerprint verify-hf [--models a,b,c] [--local-files-only]
lm-fingerprint prove [--local-files-only]
lm-fingerprint compare --a a.json --b b.json
lm-fingerprint match --fingerprint a.json [--library PATH]
lm-fingerprint drift --baseline old.json --current new.json
lm-fingerprint enroll --fingerprint a.json --stack-id ID --library PATH
lm-fingerprint export --fingerprint a.json

There is no --api-key flag. Older wireprint-fingerprint / stackprint-fingerprint JSON still loads.

Library

from lm_fingerprint import (
    StackLibrary,
    compare_fingerprints,
    fingerprint_endpoint,
    fingerprint_hf,
    match_library,
)
from lm_fingerprint.client import HttpTransport

fp = fingerprint_endpoint(HttpTransport(url, api_key=key), model="x", base_url=url)
hit = match_library(fp, StackLibrary.load())
local = fingerprint_hf("sshleifer/tiny-gpt2", local_files_only=True)

Reading a result JSON

schema_version = 1, probe_suite_version = lm-fingerprint-probes-v1.

Compare: weighted exact-match on probes that are ok and stable on both sides. Confidence is score × coverage. Library match reports certainty (exact / strong / possible / unknown) and the margin to the second neighbor.

Measured vs ModelPrint

Re-run:

lm-fingerprint prove --local-files-only

Same leftover as ModelPrint (nine probes in probes/index.js). Their README is the scoring rule: matching fingerprints prove shared infrastructure, not identity.

Write-up: docs/PROOF.md.

Honesty rules

1. A match is shared plumbing, not the same checkpoint.
2. Count vectors are a projection of (T). They are not (T).
3. The packaged library is synthetic. User enrolls authorized runs.
4. fingerprint-hf never loads weights and never calls hosted Inference.
5. Keys come from --api-key-env. Artifacts must not contain sk- or Authorization.
6. scripts/repro.py is the product gate. Do not claim green unless you just ran it.

Nearby tools

License

MIT. See LICENSE. ModelPrint tokenizer texts remain MIT (unclecode/modelprint) and byte-exact.