---

HackerSignal is a cyber threat intelligence (CTI) dataset and benchmark that links what hackers discuss in underground and security communities to the vulnerabilities those discussions concern. It aggregates 7,447,646 exact-deduplicated documents from 64 public forum and source identifiers across eight source layers, from hacker forums and exploit archives to NVD CVE descriptions, security advisories, bug-bounty reports, fix commits and CISA Known Exploited Vulnerabilities. On top of the corpus it defines three benchmark tasks with temporal train/validation/test splits: CVE linkage retrieval, 8-class exploit type classification, and CVE-disjoint temporal generalization to vulnerabilities published after training.

HackerSignal is published in the NeurIPS 2026 Evaluations & Datasets Track. Preprint: arXiv:2605.03158.

At a glance

Documents	7,447,646 (exact-deduplicated)
Sources	64 public forum / source identifiers in 8 source layers
Hacker-community layer	6,966,014 posts (largest forums: Gayanku, Antichat, Evolution, AntiOnline, HackForums)
Vulnerability reference	340,536 NVD CVE descriptions (the retrieval corpus for Tasks 1 and 3)
Other layers	Exploit archives, exploit Q&A, advisories, bug-bounty disclosures, fix commits, CISA KEV
Languages	English plus substantial non-English content: Russian (antichat, 1.82M posts), Persian (persiantools, 280K), and others; 30.2% of CVE-linked forum posts are Cyrillic-heavy
Time span	1988–2026
Splits	Temporal: test data postdate training data in every task
Metadata	Croissant JSON-LD, Datasheet for Datasets, source release manifest
License	Data CC BY 4.0 (LICENSE-DATA), code MIT (LICENSE); per-source terms in release governance

Benchmark tasks

Task	Type	Train	Val	Test	Corpus / total	Metrics
1. CVE Linkage Retrieval (CVE-R)	Retrieval	56,692	2,584	1,990	340,536	R@1, R@5, R@10, MRR
2. Exploit Type Classification (ETC)	8-class classification	64,413	4,735	1,735	70,883	Macro-F1, per-class F1
3. Temporal Generalization (TG)	CVE-disjoint retrieval	56,833	2,535	1,898	340,536	R@1, R@5, R@10, MRR

• Task 1 (CVE-R): given exploit, advisory, fix-commit or bug-bounty text, retrieve the CVE it refers to from all 340,536 NVD descriptions. Gold links come from source metadata.
• Task 2 (ETC): classify exploit text into injection, XSS, memory corruption, denial of service, file inclusion, authentication and access control, remote code execution, or information disclosure. Train < 2020, validation 2020–2022, test ≥ 2023.
• Task 3 (TG): the Task 1 retrieval problem with strictly disjoint gold CVE sets: training CVEs were published before 2022, validation CVEs in 2022–2023, and test CVEs in 2024 or later. The Task 3 test queries are a subset of the Task 1 test queries.

The baseline ladder covers BM25, four dense bi-encoders (all-MiniLM-L6-v2, all-mpnet-base-v2, BGE-base-en-v1.5, E5-base-v2) and a BM25 + dense hybrid for retrieval, and Decision Tree, TF-IDF + logistic regression, linear SVM, RNN, GRU, LSTM, BiLSTM, SecBERT, SecureBERT and CySecBERT for classification. Scores are in Baseline results below and in the paper (Tables 5–7).

Baseline results

Camera-ready numbers, rebuilt end to end from the released data on one Apple M5 Pro (PyTorch 2.14.1, MPS). Every value below is generated from results/camera_ready/results.csv; per-run seeds, model revisions and data hashes are in results/camera_ready/provenance.json. Retrieval uses zero-shot encoders over all 340,536 NVD descriptions, and MRR is MRR@10. The forum-post pilot masks every CVE string in the 1,964 CVE-linked hacker-community posts and retrieves the cited CVE from the same corpus.

Model	Task 1 R@1	Task 1 MRR	Task 3 R@1	Task 3 MRR	Forum-post pilot MRR
BM25	0.595	0.636	0.620	0.660	0.064
all-MiniLM-L6-v2	0.655	0.709	0.670	0.722	0.073
all-mpnet-base-v2	0.626	0.682	0.647	0.703	0.065
BGE-base-en-v1.5	0.653	0.714	0.667	0.726	0.078
E5-base-v2	0.649	0.710	0.663	0.721	0.072
Hybrid BM25 + mpnet	0.679	0.711	0.698	0.730	–

The top two models are statistically tied on both tasks (paired bootstrap, Task 1 ΔMRR 0.003, 95% CI [-0.013, 0.019]; Task 3 ΔMRR 0.004, 95% CI [-0.012, 0.020]).

Task 2 (exploit type classification, 1,735 test exploits). Transformers are fine-tuned on a 15K training subsample.

Model	Macro-F1 (mean ± SD, 3 seeds)	95% CI
Decision Tree	0.702 ± 0.007	[0.677, 0.727]
TF-IDF + LR	0.772 ± 0.000	[0.744, 0.795]
SVM	0.799 ± 0.000	[0.773, 0.822]
RNN	0.436 ± 0.031	[0.415, 0.455]
GRU	0.858 ± 0.001	[0.840, 0.874]
LSTM	0.824 ± 0.013	[0.804, 0.841]
BiLSTM	0.856 ± 0.003	[0.837, 0.872]
SecBERT	0.852 ± 0.007	[0.833, 0.870]
SecureBERT	0.892 ± 0.008	[0.875, 0.907]
CySecBERT	0.898 ± 0.003	[0.882, 0.913]

CySecBERT and SecureBERT do not differ significantly (paired Δ 0.006, 95% CI [-0.003, 0.016]).

Quick start

The test splits are committed in data/benchmark_v2/. Training and validation splits and the full corpus are on the Hugging Face Hub as JSONL files:

from datasets import load_dataset