NEEDLE

Search and retrieval benchmarks from Keenable.ai. A benchmark is a query set plus a scoring method. Each one is a module under needle with the same two subcommands:

needle <benchmark> generate ... --out queries.jsonl   # produce query rows (JSONL)
needle <benchmark> run --queries queries.jsonl ...    # evaluate engines on them (JSON report)

Benchmark	Queries	Scoring
news	fresh, time-sensitive queries from live RSS + Google Trends	LLM relevance judge → nDCG@5
finance	company and financial fact lookups, gold from public registries and SEC filings	answer-recall@K + MRR@K, deterministic (optional LLM backstop)
scholar	known-item paper retrieval: by title vs. by a full-text-only detail	recall@K + MRR@K by paper-id match
legal	known-item caselaw / CFR retrieval	recall@K + MRR@K by citation/docket/URL identity
agentic_rare	English rare-word queries sampled from a filtered query stream	LLM relevance judge → nDCG@5

Each bench generates gold on demand from public sources; the repo commits none of it. agentic_rare differs: it is a query producer plus a news-style eval, with no gold of its own.

Install

Uses uv: uv sync, then uv run needle ... (or uv tool install --editable . for a global needle).

Configuration

The CLI loads .env from the working directory (copy .env.example); exported variables take precedence.

Variable	Purpose
OPENROUTER_API_KEY	all LLM work — query projection (news, finance filingdoc, scholar, legal code) and judging
EXA_API_KEY, SERPER_API_KEY (google), SEARCHAPI_API_KEY (bing), BRAVE_API_KEY, PARALLEL_API_KEY, TAVILY_API_KEY, LINKUP_API_KEY, PERPLEXITY_API_KEY, OCTEN_API_KEY, CERAMIC_API_KEY, YOU_API_KEY, FIRECRAWL_API_KEY, TINYFISH_API_KEY, KAGI_API_KEY, JINA_API_KEY, CONTEXT_DEV_API_KEY (context), DEEPSEEK_API_KEY, ANTHROPIC_API_KEY (claude-search), OPENAI_API_KEY (chatgpt-search)	one per engine, required when that engine is in --engines
KEENABLE_API_KEY	optional — without it the CLI uses the keyless, rate-limited endpoint
NEEDLE_LLM_MODEL, NEEDLE_JUDGE_MODEL	both default to openai/gpt-6-luna; --llm-model / --judge-model override

All run commands share one interface:

Flag	Default	Meaning
--queries	(required)	JSONL query rows, typically from generate
--out	-	report path; - = stdout
--engines	keenable,exa	comma-separated engine list
--num-results	5	top-K fetched and scored per engine
--snippet-chars	2000	uniform cap on per-result evidence text (0 = no cap), so engines that return more text get no free evidence; also sent as the snippet-length request budget to engines that accept one (keenable, exa, ceramic), clamped to each API's range; jina has no such parameter, so the client clips the fetched page text to this budget before it reaches the report; brave-llmcontext clips its joined snippets the same way
--limit / --sample / --seed	0 / stratified / 0	deterministic sample of N queries
--judge-model / --judge-concurrency	env / 8	LLM judge knobs

Each engine has one request in flight at a time (scholar runs with --search-concurrency 2), so latency samples are comparable across engines. They time successful requests excluding connection warm-up: the requests that open an engine's first connections in a run are not timed, since they pay one-time connection setup and, where a bench queries all engines in parallel, every engine starting at once. Latency therefore shows performance on repeated requests. A later request that has to open a new connection is timed and counted in latency.reconnects. Per-engine tuning uses env vars: TAVILY_DEPTH, LINKUP_DEPTH, NEEDLE_CLAUDE_SEARCH_MODEL (default claude-sonnet-5), NEEDLE_CHATGPT_SEARCH_MODEL (default gpt-5.5); the engine entry fixes Keenable's, Exa's, Parallel's and Perplexity's modes and Brave's endpoint (keenable-realtime = realtime, exa = auto, exa-instant = instant, exa-lite = lite, parallel = advanced, parallel-fast = fast, parallel-turbo = turbo, perplexity = the default web search type, perplexity-fast = fast, brave = /web/search, brave-llmcontext = /llm/context). A failed search scores zero when the engine's side caused it (timeout, server error, content filter, a rejected query). The mean skips it only when our own account or plan caused it: HTTP 401, 402, 403, 429, 432 or 433, or a 400 or API error that names our credits, quota, usage limit, billing or API key. Clients pace requests to the vendor's rate limit (min_interval_s) and retry a 429 up to 6 times with exponential backoff, holding their concurrency slot while they wait. The mean also skips queries whose judging failed. The report lists search_errors, split into account_errors and service_errors, and judge_errors; num_scored counts the queries in the mean.

claude-search and chatgpt-search go through a model's server-side web search tool, since neither vendor exposes the search index directly. The prompt asks the model to search once and reply with a JSON array of every result it received, in order, with the snippet copied verbatim (up to 500 characters). Rank is the tool's own result order (Claude's web_search_tool_result blocks, ChatGPT's web_search_call.action.sources); the JSON supplies title and snippet for each URL, and JSON rows whose URL is not among the tool's hits are dropped. If the JSON fails to parse the hits still count, with title and page age from Claude and bare URLs from ChatGPT. Claude runs one search per query (max_uses=1); ChatGPT runs gpt-5.5 at low reasoning with the search tool forced. Output is capped at 32k tokens. The prompt also tells the model to pass the query to the search tool verbatim, operators included, so site:/after:/before: reach the underlying engine as text; site: is additionally set as the tool's domain allow-list.

news

Turns what trends right now into keyword queries; the bench scores how well each engine ranks results for them.

needle news generate --source rss --out queries.jsonl    # or --source trending
needle news run --queries queries.jsonl --limit 20 --out ndcg.json

generate builds one cohort from two sources. RSS: ~124 curated public feeds, the newest item per feed within a freshness window (1h for most feeds, 168h for papers); when too few survive, the least-stale items fill the gap. Google Trends: the keyless Trends RSS across all 52 US geos, merged, fuzzy-deduped, capped by traffic. An LLM projects each item into a query and refuses evergreen content. query_id is deterministic from (query_text, hour_ts). The feed list lives in feeds.default.toml (override with --feeds; honor each publisher's ToS and rate limits).

run judges each engine's results as returned — its own title and snippet — with an LLM relevance judge (Google "Needs Met" 0–4) and scores nDCG@5 with a redundancy penalty for duplicate URLs. The ideal ranking is the pooled ultimate ordering — the best rating per deduplicated URL across all engines, oracle-ranked — so ultimate scores exactly 1 and queries where no engine found anything relevant are excluded. The judge rates results identical across engines once. --snippet-chars keeps judge evidence uniform across engines. --queries also accepts plain text, one query per line.

finance

Keyword queries about public companies ("nvidia ceo", "nvidia q1 fiscal 2026 net income") with gold answers from Wikidata, GLEIF, and SEC filings. The score asks whether top-K results contain the answer, not whether they hit one pinned URL.

needle finance generate --max-companies 100 --per-company 1 --filingdoc-target 40 --out gold.jsonl
needle finance run --queries gold.jsonl --judge --out report.json

Three suites (--suites): finance — registry facts per company (ceo, ceo_since, ceo_company, founded_year, hq_country, website, employees, lei, ticker), each asked as a keyword query and, where a template exists, as a natural-language variant (bucket finance_nl); filings — single-quarter 10-Q facts (default net income, operating income, diluted EPS; --fields can add revenue) from SEC XBRL, stratified by market cap; filingdoc — known-item filing retrieval, gold = the accession number. ~50% of filings/filingdoc queries carry operator syntax ("quoted", site:, after:/before:), tagged in query_origin.syntax.

A deterministic matcher scores each field: nickname-tolerant person names, a 0.1% band for money, 15% for employee counts, country aliases, registrable-domain match for websites, and required cue words for low-entropy fields so a stray number cannot score. Money also matches a bare table figure in thousands or millions (801 for $801M, 185,808 for $185.8M) and a parenthesized loss ((458), $(3.98)); a year or a percentage never counts. --judge adds an LLM backstop for non-money fields, consulted only for rejected results ranked ahead of the first deterministic hit — it can upgrade a miss but never revoke a hit. The report breaks down by field, suite, syntax, cap tier, and freshness, and splits misses into system-specific vs universal. Snippet-only checks give a lower bound, the same for every engine.

scholar

Known-item paper retrieval. Each paper yields four queries: title (degraded title, answerable from metadata), body (a keyword query on a full-text-only detail, machine-verified absent from title and abstract), clue (the same kind of full-text facts phrased as a natural-language question), and tot (a tip-of-the-tongue description: hedged, half-remembered, with names and exact values banned and machine-verified absent). The gaps between buckets are the goal: title vs body isolates full-text indexing, body vs clue isolates keyword-vs-prose handling, and tot isolates semantic retrieval. A paper stays only if all requested buckets pass, so buckets remain comparable; --buckets selects a subset.

needle scholar generate --age-buckets 7d,30d,1y --per-cell 10 --out gold.jsonl
needle scholar run --queries gold.jsonl --num-results 10 --search-concurrency 2 --out report.json

Gold comes from two suites (--suites): arxiv and europepmc, paired and balanced over (domain × age) cells. Scoring is deterministic: the scorer scans result URLs and snippets for arXiv ids, DOIs, and PMIDs; a query is a hit when an id matches the gold paper. The report separates per-bucket recall (by_bucket) and splits misses into system-specific vs universal. Pages with no inline identifier cannot match, so recall is a lower bound, the same for every engine.

legal

Known-item legal retrieval; gold is a document identity from the citation graph, so it needs no expert annotation. ~50% of queries carry operator syntax.

needle legal generate --per-court 4 --per-title 4 --out legal.jsonl
needle legal run --queries legal.jsonl --out legal.json

Two suites (--suites): caselaw — recent published opinions from the CourtListener API across 14 federal courts, one caption-style query per case (gold: reporter citations, docket + party tokens, cluster id); each court searches --months-per-court (default 3) random months of the last --months-back (18), one request each, to stay under the token's 50 searches/hour; code — eCFR sections; an LLM projects each into a query about the section's substance with one verbatim quoted span, and the bench rejects it if it leaks the citation (gold: the title CFR section citation). run scans result URLs and snippets for reporter citations, docket numbers (scored only next to a gold party token), CourtListener/Justia URLs, and CFR citations; it breaks down by suite, syntax, and court.

agentic_rare

A query producer plus a news-style eval for English rare-word queries.

uv run python scripts/agentic_rare_filter.py --out rare_entity.parquet   # refresh the filtered artifact
needle agentic_rare generate --limit 100 --out queries.jsonl
needle agentic_rare run --limit 100 --out report.json

scripts/agentic_rare_filter.py reads a query stream (agentic/queries.parquet on the HF dataset) and keeps queries with at least one rare word: a word that BERT WordPiece splits into ≥5 subwords or maps to [UNK]. It drops too-short, too-long, URL-like, and non-Latin queries, plus VINs, hex hashes, and crypto addresses. fastText language ID gates language: it rejects confident non-English, accepts confident English, and otherwise requires that most non-rare words are common English (wordfreq). An n-gram cap (3-grams, max 2 each) dedups near-copies. Each kept row carries length_bucket (short/medium/long by word count) and its hard words with their subword splits.

generate and run sample the filtered artifact (agentic/rare_entity.parquet), stratified by length_bucket; run scores exactly like news (LLM relevance judge, nDCG@5). --queries reads a local file instead of the HF artifact; run --queries-out also saves the sampled query rows.

Shared infrastructure

needle.shared.search — a SearchClient protocol (errors-as-data async search(query, *, num_results) -> (results, error)) with clients for every engine above. The client parses out Google-style operators (site:, after:, before:) and translates each to the engine's best native mechanism; it drops unsupported ones rather than send them as literal tokens. To add an engine: subclass HttpSearchClient, map its response to SearchResult, add an EngineSpec to ENGINES in factory.py, and set the key env var — reports and the dashboard pick it up automatically.

Pipelines are pure (inputs + clients in, rows/reports out), so a notebook or scheduler can also drive them: the ranking harness is needle.shared.rankeval.run_ndcg, the recall scorer needle.finance.score.run_answers.

Every bench report carries a synthetic ultimate engine: the pooled results of all engines per query, oracle-ranked — by judge rating on the judged benches, gold item first on the known-item benches. It is the nDCG ideal ranking on the judged benches and the score ceiling for any single engine; the overlap and uniqueness stats exclude it.

Continuous benchmarks

bench.yaml runs against all registered engines: news hourly (--limit 20); twice daily with fresh gold — finance 00:17/12:17 UTC (--limit 120 --judge), agentic_rare 03:17/15:17 (--limit 100), scholar 06:17/18:17 (--per-cell 7), legal 09:17/21:17. Gold lives on gh-pages between runs; a manual dispatch runs only the benches it selects; a daily bench also runs at any scheduled hour when its gold file is missing. Each bench is its own job and publishes its own run, so a slow daily bench never delays the hourly news result. Each run appends summary rows (history.jsonl), engine-pair URL overlap (overlap.jsonl), and per-engine unique-URL counts (uniqueness.jsonl) to gh-pages, and rewrites independence.json with their all-runs and last-7-days totals; the dashboard at https://keenableai.github.io/needle/ renders history.jsonl and independence.json. Raw per-query latency samples (lat_ms) stay in history.jsonl for 7 days only. Each run also archives full artifacts to the HF dataset keenable-ai/needle-results (a slim <bench>.json per run plus <bench>/<engine>.json with each engine's per-query results) and refreshes daily_queries.jsonl at the dataset root: a rolling window of the queries evaluated by runs from the last 24 hours (scripts/daily_queries.py; backfill rebuilds it from the archive). The workflow needs repo secrets OPENROUTER_API_KEY, HF_TOKEN, COURTLISTENER_API_TOKEN, and the per-engine keys.

The deployed site also serves the results page at https://keenableai.github.io/needle/results/ with results.csv, results.json, results.md and results.pdf, rebuilt on every deploy, plus a sitemap.xml that lists the dashboard and the results page. snapshot.yaml runs on the 1st of each month and saves the month before as it stood at its end (the 7-day mean to its last day) under history/ on gh-pages: needle-results-YYYY-MM.{csv,json,md} and needle-overall-YYYY-MM.{webp,png}. It renders the gh-pages state from just before the next month's first run with the browser clock set to the month's end, and it never overwrites a month that already exists. The results page lists the months in its History section; add a month's blog post to dashboard/history_posts.json as {"YYYY-MM": "https://..."}.

Development

uv sync
uv run pytest   # deterministic, no network

Citation

@misc{needle2026,
  author = {Styskin, Andrey and Petri, Matthias and Gusev, Ilya},
  title  = {{NEEDLE}: A Live Open-Source Search Benchmark for {AI} Agents},
  year   = {2026},
  url    = {https://keenableai.github.io/needle/},
  note   = {Contact: [email protected], [email protected], [email protected]}
}

License

Code: MIT — see LICENSE. Benchmark results data (the dashboard data, the results page files, the monthly snapshots and the keenable-ai/needle-results dataset): CC BY 4.0.