Nuxt Evals

Agent evaluations for Nuxt coding tasks, powered by @vercel/agent-eval.

Setup

pnpm install
cp .env.example .env   # requires VERCEL_OIDC_TOKEN and AI_GATEWAY_API_KEY

Scripts

pnpm run eval

Runs agent evaluations with memoization. Only runs (model, eval) pairs that haven't been completed yet.

pnpm run eval                          # Run only missing pairs
pnpm run eval -- claude-opus-4.6       # Run a specific experiment
pnpm run eval -- --force               # Re-run everything
pnpm run eval:smoke                    # Run smoke test (1 eval per experiment)
pnpm run eval:dry                      # Preview what would run

pnpm run export-results

Exports clean results to agent-results.json.

pnpm run export-results                          # Export from all experiments
pnpm run export-results -- claude-opus-4.6       # Export specific experiment

Each experiment also gets an avgCostUsd: the mean list cost per eval. Tokens are read from each run's transcript-raw.jsonl (handled per harness in scripts/cost.ts) and multiplied by the list prices in MODEL_PRICING. A model with no price entry, or whose runs carry no token usage, exports null and renders as N/A. Prices are a snapshot, so re-run the export to refresh them.

pnpm run test:cost

Unit tests for the token extraction and pricing in scripts/cost.ts.

Models

Experiment	Agent	Model
claude-opus-4.6	vercel-ai-gateway/claude-code	anthropic/claude-opus-4.6
claude-opus-4.7	vercel-ai-gateway/claude-code	anthropic/claude-opus-4.7
claude-opus-4.8	vercel-ai-gateway/claude-code	anthropic/claude-opus-4.8
claude-opus-5	vercel-ai-gateway/claude-code	anthropic/claude-opus-5
claude-opus-5.5	vercel-ai-gateway/claude-code	anthropic/claude-opus-5.5
claude-fable-5	vercel-ai-gateway/claude-code	anthropic/claude-fable-5
claude-fable-5.1	vercel-ai-gateway/claude-code	anthropic/claude-fable-5.1
claude-fable-5.2	vercel-ai-gateway/claude-code	anthropic/claude-fable-5.2
claude-sonnet-4.5	vercel-ai-gateway/claude-code	anthropic/claude-sonnet-4.5
claude-sonnet-4.6	vercel-ai-gateway/claude-code	anthropic/claude-sonnet-4.6
claude-sonnet-5	vercel-ai-gateway/claude-code	anthropic/claude-sonnet-5
claude-sonnet-5.5	vercel-ai-gateway/claude-code	anthropic/claude-sonnet-5.5
cursor-composer-2.0	cursor	composer-2
cursor-composer-2.5	cursor	composer-2.5
deepseek-v4-pro	vercel-ai-gateway/opencode	vercel/deepseek/deepseek-v4-pro-0813
gemini-3.1-pro-preview	vercel-ai-gateway/opencode	vercel/google/gemini-3.1-pro-preview
gemini-4-argon	vercel-ai-gateway/opencode	vercel/google/gemini-4-argon
gemini-4-carbon	vercel-ai-gateway/opencode	vercel/google/gemini-4-carbon
glm-5.3	vercel-ai-gateway/opencode	vercel/zai/glm-5.3
gpt-5.3-codex-xhigh	vercel-ai-gateway/codex	openai/gpt-5.3-codex?reasoningEffort=xhigh
gpt-5.4-xhigh	vercel-ai-gateway/codex	openai/gpt-5.4?reasoningEffort=xhigh
gpt-5.5-pro	vercel-ai-gateway/codex	openai/gpt-5.5-pro
gpt-5.6-sol-xhigh	vercel-ai-gateway/codex	openai/gpt-5.6-sol?reasoningEffort=xhigh
gpt-6-astra-high	vercel-ai-gateway/codex	openai/gpt-6-astra?reasoningEffort=high
gpt-6-sol-high	vercel-ai-gateway/codex	openai/gpt-6-sol?reasoningEffort=high
gpt-6.1-sol-high	vercel-ai-gateway/codex	openai/gpt-6.1-sol?reasoningEffort=high
grok-4.7	vercel-ai-gateway/opencode	vercel/spacexai/grok-4.7
kimi-k2.6	vercel-ai-gateway/opencode	vercel/moonshotai/kimi-k2.6
kimi-k2.7-code	vercel-ai-gateway/opencode	vercel/moonshotai/kimi-k2.7-code
kimi-k3	vercel-ai-gateway/opencode	vercel/moonshotai/kimi-k3
kimi-k3.1-pro	vercel-ai-gateway/opencode	vercel/moonshotai/kimi-k3.1-pro
minimax-m2.7	vercel-ai-gateway/opencode	vercel/minimax/minimax-m2.7
minimax-m3	vercel-ai-gateway/opencode	vercel/minimax/minimax-m3

Eval structure

Each eval is a self-contained Nuxt project in evals/. Most evals provide broken or suboptimal starter code that the agent must fix — the prompt describes a symptom without revealing the solution.

evals/nuxt-000-fix-data-fetching/
├── PROMPT.md          # task given to the agent
├── EVAL.ts            # vitest assertions (withheld from the agent)
├── package.json       # Nuxt project manifest
├── nuxt.config.ts
├── tsconfig.json
├── eslint.config.mjs
├── server/
│   └── api/
│       └── greeting.ts
└── app/
    ├── app.vue
    └── pages/
        └── index.vue  # broken starter code the agent must fix

File	Purpose
PROMPT.md	The task prompt sent to the agent
EVAL.ts	Test file run after the agent finishes (withheld from agent)
package.json	Must have "type": "module" and a "build" script
Everything else	Source files the agent can see and modify

Adding a new eval

1. Create a directory under evals/ (e.g., evals/nuxt-015-my-eval/)
2. Add PROMPT.md with a vague, symptom-based task description (don't reveal the solution)
3. Add broken or suboptimal starter code in app/ for the agent to fix
4. Add EVAL.ts with vitest assertions that check for the correct fix and reject anti-patterns
5. Add package.json with "type": "module" and "build": "nuxt build"
6. Run it for tier-1 models only (see Model retention policy). Tier-2 models keep their last measurement and are not rerun

Adding a new model

1. Create a config in experiments/ (e.g., experiments/my-model.ts)
2. Add the display name to MODEL_NAMES in scripts/export-results.ts
3. Run pnpm run eval — it will automatically run all evals for the new model
4. Decide its tier and update TIER_ONE_EXPERIMENTS in scripts/tier-one.ts (see below)

Model retention policy

Mirrors nextjs.org/evals. The published board has two tiers:

• Tier 1 (current): models with a complete run of the current eval set. Tier 1 carries the latest model of each product line, and a new version moves its predecessor to tier 2 right away. Sibling lines count separately, but only flagship or coding-focused models are measured: no budget or fast variants. A tier-1 model that goes stale because the eval set moved on gets rerun, not left on old measurements.
• Tier 2 (previously measured): every other model keeps its last measured results for reference, dated by its timestamp, and is not rerun.

Tier 1 is scored over the full current eval set: an eval it never ran exports with notAvailable: true and counts as not passed until it is rerun. Tier 2 is scored over the evals it was measured on, so adding an eval doesn't lower it. An eval that changed since a model's last run keeps that last result.

TIER_ONE_EXPERIMENTS in scripts/tier-one.ts is the list, and each experiment exports a tier field. When the eval set changes, rerun only tier 1:

pnpm run eval:tier1:status   # what each tier-1 experiment is missing
pnpm run eval:tier1          # run it

Publishing to nuxt.com/evals

After running evals:

1. Export results: pnpm run export-results
2. Copy to the nuxt.com repo:

cp agent-results.json <path-to-nuxt.com>/agent-results.json

1. Commit and deploy the nuxt.com repo

Current evals

Nuxt (22)

Eval	Type	Tests
nuxt-000-fix-data-fetching	fix	Replace onMounted + $fetch with useFetch
nuxt-001-prefer-nuxt-link	fix	Replace <a href> with <NuxtLink to>
nuxt-002-state-composables	build	State management with useState composable
nuxt-003-page-meta	build	Page meta, useHead, and custom layouts
nuxt-004-error-handling	build	Error handling with NuxtErrorBoundary
nuxt-005-fix-seo-meta	fix	Replace useHead meta arrays with useSeoMeta
nuxt-006-runtime-config	build	Runtime config with public vs private keys
nuxt-007-avoid-redundant-ref	fix	Replace ref + watch with computed for derived state
nuxt-008-fix-exposed-secret	fix	Move private runtimeConfig access to server API route
nuxt-009-cache-api-response	fix	Replace defineEventHandler with defineCachedEventHandler
nuxt-010-fix-watch-fetch	fix	Replace watch + $fetch with useFetch reactive URL
nuxt-011-fix-sequential-fetching	fix	Parallelize sequential await useFetch with Promise.all
nuxt-012-nuxt3-to-nuxt4-migration	fix	Migrate Nuxt 3 directory structure to Nuxt 4
nuxt-013-prefer-nuxt-image	fix	Replace raw <img> with NuxtImg + @nuxt/image
nuxt-014-prefer-use-cookie	fix	Replace document.cookie with useCookie composable
nuxt-015-shared-source-of-truth	fix	Deduplicate app and server utils into the shared/ directory
nuxt-016-hydration-mismatch	build	Render a load timestamp without an SSR hydration mismatch
nuxt-017-shallow-reactivity	fix	Make a nested mutation reactive on useFetch shallowRef data
nuxt-018-nuxt5-migration	fix	Opt into compatibilityVersion 5 and update deprecated data fetching
nuxt-019-nuxt5-runtime-semantics	fix	Reserve client-only space with ClientOnly and target Vite environments
nuxt-020-fix-nuxt-module	fix	Fix a local module's meta, namespacing, config merging, and dependencies
nuxt-021-custom-fetch-client	fix	Replace a globalThis.$fetch override with a provided $fetch.create client

Nuxt Content (2)

Eval	Type	Tests
nuxt-content-000-navigation	build	Documentation site with queryCollectionNavigation sidebar
nuxt-content-001-data-collection	build	Data collection (type "data") with JSON files

Nuxt UI (8)

Eval	Type	Tests
nuxt-ui-000-theming	build	Theming with app.config.ts colors and semantic utilities
nuxt-ui-001-fix-raw-html-page	fix	Replace raw HTML with UHeader/UFooter/UPageHero/UPageSection
nuxt-ui-002-dashboard-layout	build	Dashboard with UDashboardGroup/Sidebar/Panel
nuxt-ui-003-fix-raw-form	fix	Replace raw form with UForm + Zod validation
nuxt-ui-004-table	build	Data table with UTable, columns, and search
nuxt-ui-005-modal	build	Modal overlay with UModal and v-model:open
nuxt-ui-006-command-palette	build	Command palette with UCommandPalette and keyboard shortcuts
nuxt-ui-007-dropdown-menu	build	Dropdown menu with grouped items, icons, and onSelect

License

See LICENSE.