---

HackerSignal is a cyber threat intelligence (CTI) dataset and benchmark that links what hackers discuss in underground and security communities to the vulnerabilities those discussions concern. It aggregates 7,447,646 exact-deduplicated documents from 64 public forum and source identifiers across eight source layers, from hacker forums and exploit archives to NVD CVE descriptions, security advisories, bug-bounty reports, fix commits and CISA Known Exploited Vulnerabilities. On top of the corpus it defines three benchmark tasks with temporal train/validation/test splits: CVE linkage retrieval, 8-class exploit type classification, and CVE-disjoint temporal generalization to vulnerabilities published after training.

HackerSignal is published in the NeurIPS 2026 Evaluations & Datasets Track. Preprint: arXiv:2605.03158.

At a glance

Documents	7,447,646 (exact-deduplicated)
Sources	64 public forum / source identifiers in 8 source layers
Hacker-community layer	6,966,014 posts (largest forums: Gayanku, Antichat, Evolution, AntiOnline, HackForums)
Vulnerability reference	340,536 NVD CVE descriptions (the retrieval corpus for Tasks 1 and 3)
Other layers	Exploit archives, exploit Q&A, advisories, bug-bounty disclosures, fix commits, CISA KEV
Languages	English plus substantial non-English content: Russian (antichat, 1.82M posts), Persian (persiantools, 280K), and others; 30.2% of CVE-linked forum posts are Cyrillic-heavy
Time span	1988–2026
Splits	Temporal: test data postdate training data in every task
Metadata	Croissant JSON-LD, Datasheet for Datasets, source release manifest
License	Data CC BY 4.0 (LICENSE-DATA), code MIT (LICENSE); per-source terms in release governance

Benchmark tasks

Task	Type	Train	Val	Test	Corpus / total	Metrics
1. CVE Linkage Retrieval (CVE-R)	Retrieval	56,692	2,584	1,990	340,536	R@1, R@5, R@10, MRR
2. Exploit Type Classification (ETC)	8-class classification	64,413	4,735	1,735	70,883	Macro-F1, per-class F1
3. Temporal Generalization (TG)	CVE-disjoint retrieval	56,833	2,535	1,898	340,536	R@1, R@5, R@10, MRR

• Task 1 (CVE-R): given exploit, advisory, fix-commit or bug-bounty text, retrieve the CVE it refers to from all 340,536 NVD descriptions. Gold links come from source metadata.
• Task 2 (ETC): classify exploit text into injection, XSS, memory corruption, denial of service, file inclusion, authentication and access control, remote code execution, or information disclosure. Train < 2020, validation 2020–2022, test ≥ 2023.
• Task 3 (TG): the Task 1 retrieval problem with strictly disjoint gold CVE sets: training CVEs were published before 2022, validation CVEs in 2022–2023, and test CVEs in 2024 or later. The Task 3 test queries are a subset of the Task 1 test queries.

The baseline ladder covers BM25, four dense bi-encoders (all-MiniLM-L6-v2, all-mpnet-base-v2, BGE-base-en-v1.5, E5-base-v2) and a BM25 + dense hybrid for retrieval, and Decision Tree, TF-IDF + logistic regression, linear SVM, RNN, GRU, LSTM, BiLSTM, SecBERT, SecureBERT and CySecBERT for classification. Scores are in Baseline results below and in the paper (Tables 5–7).

Baseline results

Camera-ready numbers, rebuilt end to end from the released data on one Apple M5 Pro (PyTorch 2.14.1, MPS). Every value below is generated from results/camera_ready/results.csv; per-run seeds, model revisions and data hashes are in results/camera_ready/provenance.json. Retrieval uses zero-shot encoders over all 340,536 NVD descriptions, and MRR is MRR@10. The forum-post pilot masks every CVE string in the 1,964 CVE-linked hacker-community posts and retrieves the cited CVE from the same corpus.

Model	Task 1 R@1	Task 1 MRR	Task 3 R@1	Task 3 MRR	Forum-post pilot MRR
BM25	0.595	0.636	0.620	0.660	0.064
all-MiniLM-L6-v2	0.655	0.709	0.670	0.722	0.073
all-mpnet-base-v2	0.626	0.682	0.647	0.703	0.065
BGE-base-en-v1.5	0.653	0.714	0.667	0.726	0.078
E5-base-v2	0.649	0.710	0.663	0.721	0.072
Hybrid BM25 + mpnet	0.679	0.711	0.698	0.730	–

The top two models are statistically tied on both tasks (paired bootstrap, Task 1 ΔMRR 0.003, 95% CI [-0.013, 0.019]; Task 3 ΔMRR 0.004, 95% CI [-0.012, 0.020]).

Task 2 (exploit type classification, 1,735 test exploits). Transformers are fine-tuned on a 15K training subsample.

Model	Macro-F1 (mean ± SD, 3 seeds)	95% CI
Decision Tree	0.702 ± 0.007	[0.677, 0.727]
TF-IDF + LR	0.772 ± 0.000	[0.744, 0.795]
SVM	0.799 ± 0.000	[0.773, 0.822]
RNN	0.436 ± 0.031	[0.415, 0.455]
GRU	0.858 ± 0.001	[0.840, 0.874]
LSTM	0.824 ± 0.013	[0.804, 0.841]
BiLSTM	0.856 ± 0.003	[0.837, 0.872]
SecBERT	0.852 ± 0.007	[0.833, 0.870]
SecureBERT	0.892 ± 0.008	[0.875, 0.907]
CySecBERT	0.898 ± 0.003	[0.882, 0.913]

CySecBERT and SecureBERT do not differ significantly (paired Δ 0.006, 95% CI [-0.003, 0.016]).

Quick start

The test splits are committed in data/benchmark_v2/. Training and validation splits and the full corpus are on the Hugging Face Hub as JSONL files:

from datasets import load_dataset

base = "hf://datasets/BenAmpel/HackerSignal/benchmark_v2"
task1 = load_dataset("json", data_files={
    "train": f"{base}/task1_cve_linkage/train.jsonl",
    "validation": f"{base}/task1_cve_linkage/val.jsonl",
    "test": f"{base}/task1_cve_linkage/test.jsonl",
})
corpus = load_dataset("json", data_files=f"{base}/task1_cve_linkage/corpus.jsonl", split="train")

print(task1["test"][0]["text"][:200], "->", task1["test"][0]["cve_id"])

Replace task1_cve_linkage with task2_exploit_type (fields text, label) or task3_temporal_generalization (fields text, cve_id) for the other tasks.

Reproducing the baselines

git clone https://github.com/BenAmpel/hackersignal.git
cd hackersignal
pip install -e ".[sentence-transformers]"

Place the train, validation and corpus files from the Hub under data/benchmark_v2/<task>/, then run:

# all three tasks; --task 1|2|3 runs one task, --skip-dense / --skip-neural shorten the run
PYTHONPATH=src python3 -m etg.benchmark.run_baselines_v2 \
    --data-dir data/benchmark_v2 \
    --output data/benchmark_v2/results.json

Neural models use Apple MPS when it is available and fall back to CPU. Dense encoding of the 340K-document corpus takes from a few minutes (MiniLM) to tens of minutes (base-sized encoders) per model on a recent GPU or Apple Silicon machine.

Repository layout

hackersignal/
├── src/etg/
│   ├── collectors/        # data collection scripts, one per source
│   ├── data/              # corpus unification, preprocessing and statistics
│   ├── benchmark/         # split construction and baseline runners
│   ├── eval/              # retrieval metrics
│   └── scraper/           # generic forum crawlers
├── scripts/               # task builders and figure scripts
├── data/
│   ├── benchmark_v2/      # committed test splits and task metadata
│   ├── stats/             # corpus statistics
│   ├── audits/            # manual-audit packets and LLM-judge artifacts
│   └── release/           # source release manifest and validation
└── docs/
    ├── datasheet.md
    └── release_governance.md

Collecting and rebuilding the data

Each collector in src/etg/collectors/ writes a {source}_posts.jsonl and {source}_cve_index.jsonl pair to data/:

PYTHONPATH=src python3 -m etg.collectors.nvd --output-dir data
PYTHONPATH=src python3 -m etg.data.unify_hacker_communities \
    --data-dir data --output data/unified_hacker_communities_neurips.jsonl
PYTHONPATH=src python3 -m etg.benchmark.build_benchmark_v2 \
    --corpus data/unified_hacker_communities_neurips.jsonl --output-dir data/benchmark_v2

Responsible use

HackerSignal is released for academic and defensive cybersecurity research. Author identifiers are pseudonymized with SHA-256, provenance is preserved for every record, and sources are released under one of four modes (redistributable, research-only, metadata/pointer, excluded). Do not use the data to fine-tune automated exploit generation, to de-anonymize forum users, or as an operational threat feed without human review. Takedown and correction requests are handled as described in docs/release_governance.md.

FAQ

What is HackerSignal? A 7.45M-document cyber threat intelligence corpus from 64 public sources, with a three-task benchmark for linking hacker, exploit and advisory text to CVE identifiers under temporal distribution shift.

Where do I download it? From the Hugging Face Hub at BenAmpel/HackerSignal. The test splits are also in this repository.

How are the labels made? From platform metadata, not manual annotation: source-provided CVE links for Tasks 1 and 3, and ExploitDB, 0day.today and HackerOne type fields mapped to eight classes by deterministic rules for Task 2. A 240-row stratified audit packet is in data/audits/.

How is leakage prevented? Every task uses a temporal split, and Task 3 gold CVE sets are disjoint across train, validation and test.

Citation

If you use HackerSignal, please cite:

@inproceedings{ampel2026hackersignal,
  title     = {{HackerSignal}: A Multi-Source, {CVE}-Linked Corpus and Prospective Benchmark
               for Cyber Threat Intelligence},
  author    = {Ampel, Benjamin M. and Samtani, Sagar},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS),
               Evaluations and Datasets Track},
  year      = {2026},
  eprint    = {2605.03158},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CR},
  url       = {https://arxiv.org/abs/2605.03158}
}

GitHub's Cite this repository button reads the same entry from CITATION.cff.

License

The benchmark data and documentation are released under Creative Commons Attribution 4.0 International (CC BY 4.0) for the sources whose terms allow redistribution; research-only and pointer-only sources follow the terms listed in docs/release_governance.md. The code is released under the MIT License.

Contact

Benjamin M. Ampel (Georgia State University, [email protected]) and Sagar Samtani (Indiana University, [email protected]). Please open an issue for bugs, data errors or takedown requests.