maureille/agentic-pcapBenchmark code and interactive results for autonomous network-attack detection by tool-using LLM agents investigating raw PCAPs.
Can a tool-using LLM agent independently investigate a raw network capture and identify an attack?
Which packet-analysis queries will it choose, and can it remain effective when the same traffic is surrounded by much more background activity?
This project evaluates whether a tool-using LLM can identify an attack directly from a raw packet capture and how its investigation changes under heavier benign background traffic. Each investigation starts with a PCAP path and generic descriptions of 11 possible attacks. The model receives no precomputed packet features or traffic summary.
flowchart TB
low["Low background<br/>11 attack captures + 11 benign captures"] --> matrix
high["Generated high background<br/>11 attack captures + 11 benign captures"] --> matrix
prompts["6 prompt variants<br/>generic labels and observable descriptions"] --> matrix
configurations["18 model/reasoning configurations"] --> matrix
matrix["44 PCAPs × 6 prompts × 18 configurations<br/>4,752 completed investigations"]
matrix -->|"PCAP path; no precomputed features"| agent["Tool-using OpenCode agent"]
agent -->|"selects a query"| tools["Packet-analysis and shell tools"]
tools -->|"returns observations"| agent
agent --> decision["One capture-level decision"]
agent --> trace["Complete investigation trajectory"]
decision --> results["Scoring and aggregate results"]
trace --> dashboard["Session dashboard"]
| Resource | Contents |
|---|---|
| Hugging Face dataset and artifact page | Canonical release with the 44 PCAPs, all 4,752 investigation trajectories, prompt variants, model configurations, aggregate results, licensing information, and citation metadata |
| Interactive results portal | Result tables, seven study figures, model comparisons, and the complete session dashboard |
This repository contains the MIT-licensed implementation and reproduction workflow. The accompanying manuscript by Matthieu Aureille and Jan Fesl has been submitted for peer review; a public link will be added when available.
| Traffic condition | Capture composition |
|---|---|
| Low background | Scenario traffic and normal testbed idle activity, without generated background traffic |
| Generated high background | The same type of scenario combined with substantially more automatically generated benign traffic |
Each condition contains 11 attack captures and 11 benign captures. Every capture has one fixed expected output.
The table provides a reader-facing interpretation of the generic benchmark
labels. Underlying attack names, scan names, and tool names were not shown to
the LLM. Prompts used only Attack 1 through Attack 11 and described
observable traffic characteristics.
In the generated prompt files, “anonymized attacks” means that attack names were replaced by generic labels; the packet data itself was not anonymized.
| Benchmark label | Indicative underlying class (not shown to the LLM) |
|---|---|
| Attack 1 | TCP ACK flood using IP fragmentation |
| Attack 2 | ARP spoofing |
| Attack 3 | DNS flood |
| Attack 4 | FIN scan |
| Attack 5 | ICMP fragmentation flood |
| Attack 6 | NULL scan |
| Attack 7 | TCP SYN port scan (nmap -sS) |
| Attack 8 | Slowloris |
| Attack 9 | UDP fragmentation flood |
| Attack 10 | UDP scan |
| Attack 11 | Xmas scan |
| Goal | Start here |
|---|---|
| Reproduce the benchmark | RUNBOOK.md |
| Understand the implementation | scripts/README.md |
| Inspect benchmark configuration and prompt variants | config/ and prompts/ |
| Review model, software, testbed, and parser metadata | reproducibility/ and vllm_parsers/ |
| Explore generated figures and dashboards | docs/ |
flowchart LR
entry["scripts/*.py<br/>command-line entry points"] --> benchmark["benchmark/<br/>execute · schedule · resume"]
entry --> reporting["reporting/<br/>score · aggregate · publish"]
entry --> orchestration["orchestration/<br/>optional remote model lifecycle"]
sources["config/ · prompts/ · PCAP workspaces"] --> benchmark
orchestration --> benchmark
benchmark --> artifacts["Retained run artifacts"]
artifacts --> reporting
reporting --> outputs["CSVs · figures · dashboards"]
Shared path, workspace, OpenCode, rendering, and result utilities live directly
under scripts/agentic_pcap/. The complete internal module map is documented
in scripts/README.md.
The Python implementation uses only the standard library. Live investigations
also require OpenCode, TShark, tcpdump, common Unix tools, the PCAP
workspaces, and a configured model provider.
[!WARNING] The agent executes unrestricted shell commands. Run it only in an isolated container or disposable virtual machine.
Follow the fresh-machine execution guide for Debian/Ubuntu setup,
hardware-specific vLLM configuration, execution, recovery, and dashboard
generation. Historical launch commands document the study environment;
config/benchmark_orchestrator.local.json is the template for a new machine.
No sampling or seed override was applied. Exact runtime metadata is retained
under reproducibility/, and prebuilt sessions and
dashboards are available from the canonical Hugging Face release.
If you use this benchmark, please cite the Hugging Face dataset and reference this GitHub code repository.
Suggested dataset citation
M. Aureille and J. Fesl, “Agentic LLMs for Network Attack Detection,” version 1.0.0, Hugging Face dataset, 2026. [Online]. Available: https://huggingface.co/datasets/maureille/agentic-pcap
BibTeX
@misc{aureille2026agentic,
author = {Matthieu Aureille and Jan Fesl},
title = {Agentic LLMs for Network Attack Detection},
year = {2026},
howpublished = {Hugging Face dataset},
note = {Version 1.0.0},
url = {https://huggingface.co/datasets/maureille/agentic-pcap}
}
Machine-readable software citation metadata for this repository is available
in CITATION.cff.
This project uses separate licenses for its software and data components: