OpenRouterTeam/benchmark-harness
OpenRouterTeam/benchmark-harness has 26 GitHub stars and 16 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-15 17:25 UTC. Last push 2026-11-13.
Measured star growth
- in 24h
- 0
- in 7 days
- Update delayed
- in 15 days
- Update delayed
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
OpenRouter TypeScript harness for reproducible LLM benchmarks and evaluations.
AIREADME.md
OriginalOpenRouter Benchmark Harness
OpenRouter's internal benchmarking harness, externalized for transparency. We port benchmarks here so we can run them scalably on our infrastructure and iterate quickly.
bun install
OPENROUTER_API_KEY=... bun run bench -- --benchmark gpqa_diamond --model openai/gpt-4o-mini --limit 5
See CONTRIBUTING.md before proposing changes. Report security issues privately as described in SECURITY.md.
Router cost tiers
Select a Jev tier with --model typesafe/jev-router --cost-tier low, medium, or high. The harness sends plugins: [{ "id": "jev-router", "cost_tier": "high" }] for high, including subsequent turns. Omitting the tier preserves the API's configured default. Cost tier and reasoning effort are separate settings.
The same --cost-tier option works with openrouter/auto and openrouter/auto-beta, using their respective auto-router and auto-beta-router plugin IDs. All three share request construction across ordinary model calls and search. Auto Router's numeric tradeoff and model-pinning controls are not sent to Jev.
To benchmark the current live policy, leave experimentIds unset. A Jev experiment selects its own routing configuration before the cost tier is resolved; combining an old experiment with costTier: "high" does not select the current live high configuration.