Cases

Each case is one fixed prompt that every agent uses verbatim, in one shot. Click a thumbnail or case name to open its blind-vote page.

Mythos Craft · Single-file HTML A playable Minecraft-style 3D voxel world mythos-craft	Pelican by the Sea · SVG The classic eval: draw a pelican riding a bicycle, in code pelican-cycling
Skeleton Watch · Single-file HTML A rotatable, exploded-view 3D skeleton watch skeleton-watch	Turf War · JS agent Territory-grabbing agents battle head-to-head, ranked by wins, not votes turf-war
CS Dust2 5v5 · React project A from-scratch 3D FPS in React + three.js cs-dust2	Zhang Beihai: Meteorite Bullet · Three.js project A long-horizon 3D film adapted from The Three-Body Problem beihai-meteorite
Water-Driven Clock Tower · 3D project A 30-second physics-driven 3D explainer of Su Song's water-powered astronomical clock tower water-clock-tower

Full prompts live in cases.json.

How it works: Arena blind battles & leaderboard

• Arena (default): each case pits two anonymous artifacts against each other — vote A / B / Neither, and identities reveal only after you vote.
• Demo Show: switch top-right to browse everything, grouped by model; a model with several harnesses can be viewed singly or side by side.
• Community leaderboard: unlocks after 3 votes, ranked by Bradley-Terry scores (the LMSYS Chatbot Arena approach); "Neither" counts separately as both-bad. Votes feed a sitewide crowdsourced pool, guarded by Turnstile.

Core idea: Model + Harness = Agent

A model is just weights; wrap it in a runtime and it becomes an agent that can actually do the work. That agent is the unit this bench evaluates.

• Model: the weights and inference engine itself — GPT-5.5, Gemini 3.1 Pro, Claude Fable 5. Reasoning effort counts as part of the model: the same weights at high vs low effort feel like two different models, so judging a model while ignoring its effort isn't fair play. This bench defaults to max and labels the effort of every combination.
• Harness: the product/scaffolding wrapping the model — Codex CLI, Cursor, AntiGravity, the Claude web app. It controls tool calls, system prompts, context management and continuation strategy, and often shapes the outcome as much as the model does.
• So a different harness means a different agent and a separate entry (GPT-5.5 Pro and GPT-5.5 running in Codex CLI at xhigh are two agents from the same model family).
• Only the out-of-the-box harness is evaluated: do not load skills / plugins / MCP servers / custom system prompts — they shift the output as much as swapping the harness does and break comparability between agents. If maintainers agree an enhanced output is still worth preserving, the run may set "evaluation": "showcase" and remain viewable only; it is excluded from Arena, voting, rankings, and qualification counts.

Registry

More than a hundred model x harness x effort combinations have run these cases, from Anthropic, OpenAI, Google, DeepSeek, xAI, Moonshot, Zhipu, Alibaba and more. Browse the artifacts at bench.nagi.fun; the full list is below.

Contributing a run

Contributions are data-only — no code changes: you add two files. Full rules live in AGENTS.md (the contributor / AI-agent guide).

Four steps:

1. Generate: run the prompt from cases.json verbatim in your harness, once (every skill / plugin disabled, no MCP, no custom system prompt).
2. Add the artifact: preferably at outputs/<base-model>/<harness-effort>/<case-id>.<ext> (e.g. outputs/gpt-5-5-pro/chatgpt-web-extended-pro/pelican-cycling.svg).
3. Register it: models/<agent-id>.json (new file for a new combination; add to runs for an existing one) with artifactDir pointing at the artifact folder (example below).
4. Check and open a PR: run bun scripts/validate-data.ts until it passes, then open a PR with the template's proof screenshot — your harness showing the model name and effort while generating (paste it into the PR; do not commit image files).

{
  "label": "GPT-5.5 Pro",
  "vendor": "OpenAI",
  "harness": "ChatGPT Web",
  "effort": "Extended Pro",
  "artifactDir": "gpt-5-5-pro/chatgpt-web-extended-pro",
  "order": 20,
  "runs": {
    "pelican-cycling": {
      "note": {
        "zh": "在哪个 Harness 里、什么思考档位、是否一次生成、是否有修复",
        "en": "Which harness, what effort level, one-shot or fixed"
      },
      "contributor": "your-github-username"
    }
  }
}

Rules (enforced by CI — the build fails otherwise):

• agent-ids use dash-only lowercase letters / digits (e.g. doubao-seed-2-0-pro) and are stable identities; without artifactDir, artifacts default to outputs/<agent-id>/;
• artifactDir uses one or two lowercase dash-case path segments (preferably <base-model>/<harness-effort>); it is only the file location and does not affect vote or leaderboard identity;
• a bilingual provenance note is required per run — it is where this repo's credibility comes from (which harness, what effort, one-shot or fixed);
• every declared run must have its artifact;
• multiple versions of the same combination x case go in an array under runs.<case-id>, and every version after the first needs a distinct file name (e.g. pelican-cycling-2.svg);
• set contributor to your GitHub username — the site shows your avatar linked to your profile next to the run;
• skill/plugin/non-standard outputs require maintainer approval plus "evaluation": "showcase", must disclose the setup in both notes, and stay permanently excluded from blind voting and rankings;
• fill in harness and effort truthfully for new combinations: the case page renders Harness / Effort metadata chips from these fields and auto-matches brand icons for the model, vendor and harness (via lobe-icons) — contributors never touch icons.

PRs get data validation from CI; merges to main rebuild the site automatically (usually instant, at most 6 hours).

outputs/<artifact-dir>/<case-id>.<ext>  raw model artifacts (HTML / SVG), filename = case id
models/<agent-id>.json                  agent registry: label / vendor / harness / effort / artifactDir / order / run notes
cases.json                              case definitions (bilingual prompts, maintainer-owned)
scripts/validate-data.ts                data validation (CI runs it on every PR)
scripts/update-registry.ts              README Registry table generation (CI auto-syncs after merge)

bun scripts/validate-data.ts   # the same data validation CI runs (run this before submitting)
bun scripts/update-registry.ts # optional: preview the Registry tables locally; CI syncs them after merge

License

• Code (scripts/ and other code files): MIT
• Benchmark data and artifacts (cases.json, models/, outputs/): CC BY 4.0. Please credit "NAGI BENCH (https://bench.nagi.fun/)".