Jev IDS

Intrusion detection in one request. Show TypeSafe's Jev one network flow and five labeled examples. It answers whether the flow is an attack and which kind, in half a second, with no text to parse.

Jev IDS was tested on NSL-KDD, a reference benchmark of the cybersecurity community, against a state-of-the-art LLM (Gemini 3.6 Flash on Vertex AI), a classic machine-learning model (Random Forest) and an unsupervised one (Isolation Forest): 2,000 flows, three seeds, k from 0 to 8 examples per category. Given the same five examples (k = 1), Jev IDS was:

• 7.7× faster than the LLM: 0.32 s against 2.42 s per flow.
• 22× cheaper than the LLM: $74 against $1,651 per million flows, at list prices.
• As precise as the LLM: 0.953 against 0.942.
• 18× fewer false alarms than the Random Forest: 43 against 764 on the same 874 benign flows.

The LLM is the better detector on F1: 0.880 against Jev's 0.856 at k = 1, and ahead at every k (McNemar p < 0.001), because it catches more attacks of the kinds shown in the examples. Jev is the second best, and on zero-day attacks, kinds absent from the examples, it catches more than the LLM from k = 1 to k = 8 (0.747 against 0.713 at k = 1). The numbers, the figures and their limits are in Evidence and limits and docs/results.md.

Read the loop · The request template · Glossary

How it works

Jev is a System One Model. Instead of writing text, it reads a state and answers typed questions about it with probabilities. Jev IDS puts one flow into the state, next to the instructions, the column names, the five category descriptions and the labeled examples, and asks two questions. is_attack comes back as a probability. category comes back as one of five options with a confidence. The verdict is attack when the probability reaches 0.5.

Every request follows the same path:

The whole request is one file, prompts/nsl-kdd/jev.json. Python adds only the flow and the examples, and the file's sha256 travels in every prediction row as prompt_hash, so runs that asked different things are never compared as equals. Examples are labeled by category only: attack names such as neptune never reach a model.

The same flows, examples and 0.5 cut go to two baselines: an LLM through an Agno agent with a JSON output schema (Gemini 3.6 Flash through Vertex AI in the paper; GPT-5.6 through the ChatGPT Codex backend or DeepSeek as alternatives) and a scikit-learn Random Forest trained on the same examples. A third baseline, an Isolation Forest fitted on the benign flows of the pool alone, never sees an example: it is the unsupervised reference, at k = all only.

Try it

git clone https://github.com/jev-sec/jev-ids.git
cd jev-ids
uv sync
# .env: TYPESAFE_API_KEY for Jev. Gemini needs `gcloud auth application-default login` and GOOGLE_GENAI_USE_VERTEXAI=true,
# GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION=global in the environment; DEEPSEEK_API_KEY and CHATGPT_CLIENT_ID only for the other LLMs.

Download NSL-KDD into data/raw/nsl-kdd/ and prepare it once:

uv run python -m scripts.prepare_nsl_kdd
uv run jev-ids run --dataset data/nsl-kdd/dataset.json --detector jev --split smoke --k 0,1

The smoke split is five flows. The run writes results/<timestamp>-nsl-kdd-jev-smoke/ with config.json, one JSON row per flow in predictions.jsonl (the verdict, the truth, p_attack, the category, the confidence, the latency and the gateway's token report) and the raw answers in responses.jsonl.

Compare detectors

uv run jev-ids run --dataset data/nsl-kdd/dataset.json --detector jev --split pilot --k 0,1,2,4,8 --seeds 0,1,2
uv run jev-ids run --dataset data/nsl-kdd/dataset.json --detector llm:gemini --model gemini-3.6-flash --split pilot --k 0,1,2,4,8
uv run jev-ids run --dataset data/nsl-kdd/dataset.json --detector random_forest --split pilot --k 1,2,4,8,all
uv run jev-ids run --dataset data/nsl-kdd/dataset.json --detector isolation_forest --split pilot --k all
uv run jev-ids metrics results/<run_id> [results/<run_id> ...] > results/summary.csv
uv run jev-ids compare --a results/<jev_run> --b results/<rf_run> --subset novel --k-a 0 --k-b all  # novel = zero-day attacks

k is the number of labeled examples per category. k = 1 with five categories means five examples, and all means the whole pool. metrics prints one CSV row per detector and k with F1, recall on zero-day attacks (novel in the code) and per category, PR-AUC and ROC-AUC of p_attack, tokens, latency and cost. compare pairs two runs flow by flow and runs McNemar's test on the discordant pairs, because only the flows two detectors disagree on tell them apart. Cost is computed offline as tokens times the list prices in prices.json, for every detector alike.

Reproduce the paper

docs/protocol.md fixes the inputs (the 2,000-flow paper split, the prompts, the model ids, the prices) and make paper runs the four detectors over that split into results/paper/ and writes results/paper/summary.csv. Each detector has its own target (make paper-jev, make paper-llm, make paper-random-forest, make paper-isolation-forest), so they can run in separate terminals; the LLM is the slow one. make paper-llm runs Gemini 3.6 Flash on Vertex AI as five parallel processes, one per k (make -j5 paper-llm); a quota error or a timeout is an error row, and uv run jev-ids redo-errors results/paper/<run> judges those flows again in place. The full tables and figures are in docs/results.md.

Why it is fast and cheap