Table of contents

TOC

• ⚡ Getting started
  • Install skills
  • Develop from source
• ✨ Features
  • Screenshots
• ⚙️ How it works
  • End-to-end flow
  • Resume an interrupted Attempt
  • Core base
  • Plugins
• 📁 Project structure
• 📖 Docs

ageval combines programmable task scripts with declarative plugin composition, and records each step of a long-running Attempt in an activity log and checkpoints so an interruption resumes that same Attempt. Install the CLI and skills so a coding agent can author or convert a benchmark and run evaluations end to end; publish datasets, plugins, Agent packages, and evaluation results to the Hub for reproducible comparison and reuse.

Framework: datasets and tasks, plugin composition, environments, and durable Attempt state

Getting started

uv tool install ageval-cli
# install everything, or only what you need
uv tool install 'ageval-cli[all]' # every CLI extra (not the Hub service)
uv tool install 'ageval-cli[e2b]' # one extra at a time

ageval -V

Run a dataset directly from the Hub (ageval.zjureal.com, the CLI default) or from any local dataset directory:

ageval registry list                        # datasets visible on the Hub
ageval run official/[email protected] --task terminal-jsonl-agg
ageval run <org>/<name>@<version> --task <task-id>
ageval executors -v
ageval view <org>/<name>@<version> --no-browser

# continue one Attempt that stopped before result.json, or a fatal Agent invoke ERROR
ageval run <dataset> --task <id> --resume <attempt_id>
# a suite recovers an unfinished slot in place
ageval run <dataset> --resume-suite <suite_run_id>
# list retained cuts, then start a new Attempt from one of them
ageval checkpoints <dataset> --run <attempt_id>
ageval run <dataset> --task <id> --fork <attempt_id> --checkpoint <seq> --model <model_id>
# re-execute retained terminal actions in a fresh environment
ageval replay <dataset> --run <attempt_id> --evaluate

Install skills

Install skills for your local coding agent (CLI usage, plugin authoring, dataset conversion, and SDK usage):

# install all
npx skills add ZJU-REAL/ageval
# install specific ones
npx skills add ZJU-REAL/ageval --skill ageval-cli

Develop from source

To run the in-repo dataset examples and Agent packages, or to build from source, clone the repository:

git clone https://github.com/ZJU-REAL/ageval.git
cd ageval
uv sync --frozen --all-packages
uv run ageval -V

Run the minimal example and inspect the results in the local Viewer:

uv run ageval tasks examples/datasets/minimal-demo
uv run ageval run  examples/datasets/minimal-demo --task terminal-jsonl-agg
uv run ageval view examples/datasets/minimal-demo --no-browser

Features

Swap the evaluated agent and environment via plugins

Environments and agent executors both connect through plugin slots. Coding agents connect by default over ACP (pi, codex, claude-code, opencode), while nooa, dsh, miniswe, and OpenHands use the same plugin mechanism. Update profiles.yaml or pass --agent (and optional --model) to switch configurations without modifying task scripts.

Automated benchmark conversion and inspection

With the CLI and skills installed, a coding agent can author or convert a dataset and execute evaluations end to end. Afterwards, ageval view opens the dataset (or a directory of datasets) to inspect phase timings, tool-call trajectories, and reproduction commands locally, while ageval results regrade re-runs grading on archived artifact snapshots without repeating model inference.

Resume a long-running Attempt from the last completed step

ageval appends each step to an activity log and checkpoints the workspace and agent home before that step's effect settles. An Attempt that never wrote result.json continues in place with --resume, or inside a suite with --resume-suite. A run-phase billing, auth, or unrecovered transport failure seals as ERROR (agent_invoke_failed) without running evaluator.py; its recovery bundle stays, and an explicit resume rewrites that result in the same Attempt. Other settled Attempts stay immutable. --keep-recovery-state keeps the activity log, checkpoints, and replay-input.json after an ordinary settlement so ageval run --fork can start a new Attempt from one cut, under another model, with the source harness, and ageval replay can execute the recorded terminal actions again.

Isolated phase boundary and flexible grading

Only the independent evaluator.py in the evaluate phase can return PASS or FAIL, after solver sessions are sealed and reference materials (gold) are uploaded:

• Deterministic scripts: compare candidate artifacts or environment states against reference materials, optionally returning structured checks.
• Agent-as-judge / LLM-as-judge: bind an independent judge profile in profiles.yaml to inspect artifacts across single or multiple steps.

Share and compare on Hub

Publish datasets, plugins, Agent packages, and suite results to ageval Hub, or share a read-only snapshot link /s/{token} (frozen or --live) directly from a local run. Leaderboard entries record the bound harness, model, and environment and link directly to per-Attempt configurations and trajectories.

Screenshots

Leaderboard: scores by Agent and model

Trajectory: replay tool calls in the local viewer

More screenshots

Agents: packages you can pull from Hub

Plugin marketplace: environment and agent runtime plugins

Pareto: pass rate vs time

Verifier: deterministic script checks

How it works

End-to-end flow

1. ageval lock resolves the plugin dependency graph (ExtensionGraph), validates capability dependencies and ModelSpec contracts, and writes lock.json (storing credentials only as environment variable locators).
2. ageval run provisions the selected environment (local, docker, e2b, ssh, or daytona) and uploads task input data.
3. run.py drives the task interaction loop inside the run phase; switching environments or harnesses requires no changes to task code.
4. evaluator.py runs in the evaluate phase after solver sessions are sealed and reference materials (gold) are uploaded, producing the sole PASS or FAIL verdict while cleanup always executes in finally.

%%{init: {"theme":"base","themeVariables":{"actorBkg":"#eaf1ff","actorBorder":"#5B7BFF","actorTextColor":"#10233f","actorLineColor":"rgba(45,49,66,0.2)","signalColor":"#4f5d75","signalTextColor":"#2d3142","labelBoxBkgColor":"#eaf1ff","labelBoxBorderColor":"#5B7BFF","labelTextColor":"#10233f","noteBkgColor":"#f1f5f9","noteBorderColor":"#64748b","noteTextColor":"#1e293b"}}}%%
sequenceDiagram
    autonumber
    actor u as you
    participant r as run entry
    participant e as Environment<br/>local / docker / e2b…
    participant t as run.py + Agent
    participant v as evaluator.py

    u->>r: ageval lock<br/>dataset + profiles → lock.json
    Note over r: contract or capability mismatch fails fast before execution
    u->>r: ageval run
    r->>r: mint run identity
    r->>e: open environment
    r->>e: upload task data
    e->>t: execute run.py
    loop task loop
        t->>t: ACP invoke · attach_stdio
    end
    t-->>r: trajectory.jsonl
    Note over r,v: solver sealed → upload reference materials (gold), then evaluate
    r->>v: run evaluator.py
    v-->>r: PASS / FAIL / ERROR
    r->>r: record · finally cleanup<br/>lock.json · result.json · trajectory.jsonl

Resume an interrupted Attempt

A long task spends many model and tool turns inside one Attempt. If the parent process is killed before result.json is written, those completed turns and the workspace are still on disk. ageval appends every step to .ageval/runs/<attempt_id>/activity.jsonl and stores a checkpoint of /attempt/workspace and /attempt/home when the step settles.

--resume continues that Attempt when result.json is absent and the locked configuration digest still matches. It also continues a sealed agent_invoke_failed ERROR whose recovery bundle was retained; evaluate did not run for that error, and the retry restarts the failed invoke from its beginning. --resume-suite does the same for an unfinished suite slot and a recoverable invoke ERROR, while still skipping slots that settled as PASS, FAIL, or another ERROR. If the environment is still running the Attempt, the command exits without writing result.json.

A live environment that still matches the last completed step is reused. A lost environment is started again and restored from the latest checkpoint. openai-http, anthropic-http, miniswe, and terminus-2 rebuild the next model context from the activity log. acp, OpenHands, and dsh resume the session stored under the attempt home. nooa has no native session and continues from the logged transcript. Calls that already completed are not executed again, and the wall-clock time already spent stays charged. A fatal invoke does not permanently consume agent_invocations; its retry occupies one logical slot, so repeated failed resumes do not exhaust the quota. Other settled Attempts are immutable. --retry-errors continues an agent_invoke_failed ERROR in the same Attempt and allocates a new Attempt for every other ERROR; --replace-slot always allocates a new Attempt. Same-Attempt continuation overwrites the current result and suite row without adding a sample or previous[] entry.

--keep-recovery-state keeps activity.jsonl, checkpoints/, and replay-input.json after result.json and trajectory.jsonl exist. ageval checkpoints lists those cuts. ageval run --fork --checkpoint --model starts a new Attempt from one completed run-phase cut, on a fresh environment, with the source harness and a new solver model. ageval replay re-executes the retained terminal actions in another fresh environment and does not call the solver model. --evaluate grades the new artifacts. The parent Attempt stays sealed. Default pass@k skips both the fork child and the replay child.

Core base

ageval Core coordinates a fixed pipeline: lock → environment → run → evaluate → record, with resource cleanup in finally. A fatal Agent invoke ends run as ERROR and skips evaluate, so provider failures are not scored as capability FAIL. During ageval lock, Core binds the environment plugin and per-role executor plugins and validates their contracts. At runtime, Core enforces limits (wall-clock timeouts, invocation quotas, and concurrency caps) prior to execution and archives structured evidence (lock.json, result.json, trajectory.jsonl, summary.json, and task-artifacts/). An Attempt that has not settled also keeps activity.jsonl and checkpoints/ so --resume can continue it; a recoverable agent_invoke_failed ERROR keeps them for the same reason. After an ordinary settlement, --keep-recovery-state keeps that bundle so a later --fork or ageval replay can read it.

Plugins

Core decouples runtime responsibilities through exclusive slots, chain slots, and declarative dependency injection (inject). Plugins declare the slots they implement and the service capabilities they require; ageval lock resolves and verifies the dependency graph (ExtensionGraph) before execution.

# Agent executor plugin (e.g. dsh)
plugin_id: dsh
slots:
  exclusive:
    - id: executor
inject:
  - service: environment
    capabilities: [exec, upload]

# Environment plugin (e.g. docker)
plugin_id: docker
slots:
  exclusive:
    - id: environment

Project structure

ageval/
├── src/ageval/
│   ├── cli/                         # argv, help, exit code
│   ├── application/
│   │   ├── composition.py           # sole production wiring; CLI imports build_* here
│   │   ├── lock.py                  # load_and_lock
│   │   ├── run.py                   # mint identity → run_attempt
│   │   ├── regrade.py               # artifact-based regrading (skips run phase)
│   │   ├── campaign.py / suite/     # matrix · suite · Always-k
│   │   └── agent_ops/ / plugin_ops / registry_ops/ / analysis/
│   ├── attempt/                     # Attempt pipeline
│   │   ├── __init__.py              # run_attempt & run_regrade
│   │   └── phases/                  # environment → run → evaluate → record · cleanup
│   ├── config/                      # dataset + task.yaml + profiles
│   ├── model/                       # ModelSpec · resolve_model
│   ├── environments/protocol.py     # EnvironmentProvider · capabilities
│   ├── plugins/
│   │   ├── base.py / decorators.py  # BaseEnvironment · BaseExecutor
│   │   ├── slots.py                 # exclusive / chain slots
│   │   └── contrib/                 # acp · local · docker/attempt · e2b · daytona · ssh · openai_http · anthropic_http
│   ├── runtime/                     # identity, parent Agent Service, task_worker
│   ├── evaluation/                  # bind_evaluation
│   └── evidence/                    # .ageval/runs layout, activity log, checkpoints, trajectory sealing
├── src/ageval_sdk/                  # ageval_sdk for run.py and evaluator.py
├── plugins/                         # external ageval.plugin/1 (nooa, dsh, miniswe, openhands, acp-oneshot, …)
├── examples/
│   ├── datasets/
│   │   ├── minimal-demo/            # terminal-jsonl-agg · tau2-dialog-min · multiagent-env-min
│   │   ├── concurrent-research/     # concurrent-session fanout example
│   │   └── tau3-airline-5/          # airline-00 … airline-04
│   └── agents/                      # ageval.agent/1 packages
├── apps/shared                      # shared Hub + Viewer UI components
├── apps/viewer                      # ageval view SPA
├── apps/hub                         # Hub SPA
├── services/registry/               # package + results HTTP service & hosted worker
├── docs/                            # mechanism design specifications
└── website/                         # bilingual product documentation

Docs

• Usage: docs site (source)
• Design: docs/
• Examples: examples/README.md
• AGENTS.md
• ARCHITECTURE.md