Skip to content
Projects

OpenRouterTeam/benchmark-harness has 26 GitHub stars and 16 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-15 17:25 UTC. Last push 2026-11-13.

How these counts are measured

Measured star growth

in 24h
0
in 7 days
Update delayed
in 15 days
Update delayed
in 1 month
Update delayed

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

OpenRouter TypeScript harness for reproducible LLM benchmarks and evaluations.

AI

README.md

Original

OpenRouter Benchmark Harness

OpenRouter's internal benchmarking harness, externalized for transparency. We port benchmarks here so we can run them scalably on our infrastructure and iterate quickly.

bun install
OPENROUTER_API_KEY=... bun run bench -- --benchmark gpqa_diamond --model openai/gpt-4o-mini --limit 5

See CONTRIBUTING.md before proposing changes. Report security issues privately as described in SECURITY.md.

Router cost tiers

Select a Jev tier with --model typesafe/jev-router --cost-tier low, medium, or high. The harness sends plugins: [{ "id": "jev-router", "cost_tier": "high" }] for high, including subsequent turns. Omitting the tier preserves the API's configured default. Cost tier and reasoning effort are separate settings.

The same --cost-tier option works with openrouter/auto and openrouter/auto-beta, using their respective auto-router and auto-beta-router plugin IDs. All three share request construction across ordinary model calls and search. Auto Router's numeric tradeoff and model-pinning controls are not sent to Jev.

To benchmark the current live policy, leave experimentIds unset. A Jev experiment selects its own routing configuration before the cost tier is resolved; combining an old experiment with costTier: "high" does not select the current live high configuration.