relai-ai/agent-eval-loop

Eval-driven improvement for AI agents.
A plugin for Claude Code, Codex and other coding agents, by RELAI (Reliable AI).

Install · Use cases · How it works · Why RELAI · FAQ · Design

Two ways to use it

🤖 AI agents you build

https://github.com/user-attachments/assets/53ce7659-21f8-4f42-897e-a5e3cc4d1ffa

The agents you build and deploy, on any framework. Open your agent's repository and go from "it seems to work" to measured improvements in three requests:

1. Discover where it can fail

Use RELAI's discover skill to read this agent's code, prompts and the traces
in ./traces, and tell me where it is likely to fail

You get an EvalsWiki in relai/wiki/: what the agent must do, ranked risks with evidence, and the open questions only you can answer.

2. Turn the risks into evals

Use RELAI's eval skill to build tests for the top five risks and get a baseline

You get realistic tests that run the real agent end to end, with graders you preview before they are frozen, and a baseline score for each risk.

3. Improve without regressions

Use RELAI's improve skill to fix the failing tests without breaking the passing ones.
You may change prompts and tools, not the model

You get a branch with the change, before-and-after scores on the same frozen tests, and a record of what was tried and rejected.

Then keep it going as new traces arrive:

Use RELAI's loop skill to keep improving this agent from ./traces;
optimize task success first, then cost

🏭 Coding agents your team uses

https://github.com/user-attachments/assets/c8002907-e339-4679-9328-246a66a2bea1

Claude Code, Codex, OpenHands or your own setup of one. Find out whether a new model, CLAUDE.md or skill actually helps on your code, not on a public benchmark. From an eval project (a repository that holds the evals):

1. Turn past work into evals

Use RELAI's coding-evals skill to build 20 evals from the last six months of merged pull requests and reverts in the repo {name-of-the-repo} 

You get complete evals in Harbor format, ready to run. Each starts at the commit before the real fix, ships the repository and its toolchain, and is graded by hidden tests and checks the real fix must pass. Any coding agent can run them.

2. Evaluate your coding agent

Use RELAI's coding-evals skill to evaluate Claude Code on these evals and report
the solved rate and cost per solved task

3. Improve your setup before you roll it out

Use RELAI's improve skill to improve our CLAUDE.md and skills against these evals

You get a branch with the improved setup, the solved rate with its noise before and after, cost per solved task, and the evals where the two setups differ.

You can start at any step without completing the full workflow.

Installation

RELAI supports Claude Code, Codex, and other coding agents. Native installs for Claude Code and Codex are below; expand Other agents, Skills CLI, and manual installation for additional options.

Claude Code

From a terminal:

claude plugin marketplace add relai-ai/agent-eval-loop
claude plugin install relai@agent-eval-loop

Codex

From a terminal:

codex plugin marketplace add relai-ai/agent-eval-loop
codex plugin add relai@agent-eval-loop

Start a new session after installing. Codex asks you to review and trust the optional hooks before they run.

npx skills add relai-ai/agent-eval-loop --skill '*'

agy plugin install https://github.com/relai-ai/agent-eval-loop.git

gemini skills install https://github.com/relai-ai/agent-eval-loop.git --path skills

git clone https://github.com/relai-ai/agent-eval-loop.git ~/agent-eval-loop
mkdir -p .agents/skills
cp -R ~/agent-eval-loop/skills/* .agents/skills/

First steps

Not sure where to begin? Ask "Use RELAI's start skill to look at this agent and suggest a first step." It reads your repository and recommends one, without running your agent or spending on models. What each first request needs and leaves behind:

Updating

Update using the same method and scope you installed with, then start a new session. Read release notes for migration steps.

The Skills CLI and manual copies need explicit updates. Native refresh policies are described in the Claude Code and Codex docs; commands were checked with Claude Code 2.1.284 and Codex CLI 0.160.0 on 2026-11-11. Internal installs need repository read access. main can contain unreleased changes; pin a tag where supported for a fixed release.

Plugin updates do not replace existing project files in relai/ or frozen tasks. Use the project-copy policy to preserve adaptations and evidence.

Hooks

The native Claude Code and Codex installs include optional hooks, active once the plugin is enabled (Codex asks you to trust them first). They do nothing outside a project with a relai/ folder. Before an edit to a frozen test, Claude Code asks you; Codex cannot ask from a hook, so it warns the coding agent instead. After an edit under relai/, the project's checkers run and report problems to the agent. To turn them off, decline to trust them in Codex, or disable the plugin (claude plugin disable relai@agent-eval-loop); Claude Code's disableAllHooks setting also works but turns off every hook. The guarantees they surface still hold through the checkers.

Uninstalling

Commands were checked with Claude Code 2.1.322 and Codex CLI 0.167.1. Uninstalling leaves your projects' relai/ folders untouched; they keep working without the plugin.

Removing RELAI from a project. Remove candidate worktrees first (git worktree list, then git worktree remove for those under relai/work/candidates/, and git worktree prune), delete review branches you no longer want (git branch --list), then delete relai/. Keep relai/records/, changes.md and decisions.md elsewhere if you want the history of what was approved and tried. Container images built for tests can be removed with your engine's image commands.

Why RELAI

• Evals are the hard part. RELAI builds them. Evals come from traces, or are generated from what the agent must do and where it can break, including risks that never appeared as failed runs. You review them; you do not write them.
• Realistic, long-horizon environments. Evals are Harbor-format tasks: the agent runs end to end in a container, with its tools, data and as many steps as the work takes. Your project's own runner works too.
• Scores you can inspect. Code checks and LLM judges are reviewed and frozen before the first run. RELAI reports variation, supports optional held-out evals and separates infrastructure or judge errors from agent results.
• The same evals support improvement and comparison. Run the current agent, record a baseline and compare candidate changes, agents, models or setups against the same frozen evals.
• You stay in charge. You approve the risks, evals, graders, objective and budget. RELAI asks only when your judgment is needed.

How it works

When RELAI-managed state is useful, relai/STATUS.md shows the current request, relevant progress, what is waiting on you and the recommended next step.

Consequential open questions are brought to you for resolution or explicit deferral, with what their answers could change. Review handoffs offer a guided chat walkthrough or a suitable app view of the evidence and decisions; chat, app and status board use the same underlying records. When an app needs additional annotation or review capabilities, RELAI can offer to extend it.

RELAI checks the prerequisites needed for the task, names missing credentials and reuses existing runners and tests. It uses your project's credential mechanism or relai/.env; coding agents never read credential values, while executable helpers use them for authentication and redaction.

AI agents you build

Discover from code, documentation and traces

discover builds the EvalsWiki: what the agent must do, where it can fail and what remains unclear.

Point it at production traces and the coding agent can use focused checks, existing analytics, direct reading or scalable model triage for the question and budget. External processing requires appropriate data and cost authorization.

The optional triage example groups leads into three lanes:

• Needs your judgment: likely failures whose correct outcome is unclear. Your answer can become a regression eval or risk evidence.
• Coverage gaps: situations real users encounter that no eval covers, ranked by frequency.
• Weak signals: small slips that did not fail a run, informing harder or compound eval proposals.

Durable findings update EvalsWiki when they help future work. The wiki is not a transcript archive or a checklist of artifacts.

Review traces in an app

Use RELAI's trace-review skill to build a review app for ./traces
with hotkeys, filters and feedback on individual steps

The coding agent reuses your project's stack and trace tooling, then builds and verifies an app you own. Browse complete interactions, search and filter runs, leave feedback, and resume where you stopped. Traces stay read-only and data stays local by default. The app includes its own run instructions and inspectable feedback storage; no eval setup or model access is required. Source-linked feedback can later inform discovery, eval proposals or judge calibration.

Bring your own benchmark

Use RELAI's eval skill to bring in the benchmark at {path-to-benchmark}
(its own runner and scorer, train/validation/test)

RELAI first checks a small pilot against the benchmark's own scorer. It respects the benchmark's splits: fixes are learned from train, validation decides what is kept, and test is reserved for final assessment. The benchmark is pinned to its commit and never edited.

Coding agents your team uses

The coding-evals skill turns your team's work into long-horizon tasks that any coding agent can run:

Use RELAI's coding-evals skill to make a test from PR #412 in {repo-name}
Use RELAI's coding-evals skill to check whether our new CLAUDE.md helps on our tasks

• From your real work. Build tasks from a corrected coding session, pull request, revert or past fix in git history. Each task begins at the commit where the work started; the real fix is kept out of the workspace.
• Any agent can run it. Harbor installs the agent under test; the task brings the repository and its toolchain.
• Your standards. Draft "good" from CLAUDE.md, AGENTS.md, contributing guides, lint configuration and past review comments; you confirm it.
• Hard to game. Use hidden tests, existing tests as regression checks, and checks for edited tests, out-of-scope changes, unsafe work and claims that do not match what happened.
• Comparable results. Report solved rate with its noise, cost per solved task and the tasks that separate the agents or setups being compared.

Coding-agent evals live in an eval project: a repository whose relai/ holds the tasks. Task source can come from the current or other scoped repositories; each evaluated workspace contains only its starting history.

Improve and repeat

An iteration begins with new traces, answers, tests or existing evidence. The coding agent adapts, reorders, combines or skips activities to fit the request:

1. learn    triage new traces → EvalsWiki + proposed evals
2. measure  establish or reuse comparable evidence for the objective and guardrails
3. explore  investigate, measure or make a scoped change
4. check    run scope-based checks and optional held-out evals; review the results
5. record   what was learned, results, fixes kept or rejected, and spend

In practice:

• Not every iteration ships a change. An experiment can just investigate or gather evidence.
• The yardstick holds still. While an iteration runs, the tests, graders and run conditions do not change, so before and after numbers stay comparable. If a test itself turns out to be wrong, it is repaired openly, recorded, and the baseline is measured again.
• Your limits hold. The agreed budget and scope apply throughout. The coding agent asks before going beyond them or making a consequential choice you have not covered.
• Faster tooling is not a faster agent. Speeding up the test runner is reported separately from changes to the agent.

The results are evidence on the tests you have, not proof that the agent is correct everywhere or that no regression is possible.

You remain in control

relai/STATUS.md can show:

• what is waiting on you and RELAI's recommended answers;
• every risk and the eval that probes it;
• recent runs, with flaky ones flagged;
• how often each LLM judge agrees with you;
• spend against the approved budget;
• iterations, observed results and the decisions still needed from you.

Questions wait for a natural decision point: the end of discover; the start of an eval build, its scenario approval and grader preview; coding-evals scope, task and check approval and the budget before the first paid run; the first time LLM judges are scored; or a loop iteration's summary. You can hand build approvals to an independent review instead. Only a blocking safety call or conflicting requirement interrupts dependent work. The full list is in the project contract.

Built to be inspected

• Evals are frozen before comparison, with fresh-context reviews of the harness, evals and changes.
• Measurement effort is chosen for the decision; "too few attempts to tell" is a valid result.
• Optional held-out evals can be scored without reading their contents while improving the agent.
• Infrastructure and judge errors are reported separately from agent results.
• You preview how each judge grades before freezing it, and agreement is tracked.
• Tests use shared metadata and result contracts through Harbor or a native runner, with instructions for rerunning the suite, including in CI.

Requirements

• Python 3.11+ for bundled checkers and reference scripts; project-native integrations are supported. The optional hooks run the python3 first on your PATH: with an older one (such as macOS's built-in 3.9) frozen tests are still guarded, but edits are not checked automatically, and the hook says so once.
• Harbor execution: uv and a locally supported container engine with a Docker-compatible CLI, or Podman. Tasks with a restricted network also need the engine's image builder (docker buildx on a Docker CLI).
• Native execution: the project or benchmark runner's own prerequisites; unused Harbor or container tools are not required.
• Model access for the agent, simulated users and judges, through your project's credential mechanism or relai/.env.

The generated relai/README.md lists the prerequisites for each runner (see the bundled contract).

Glossary

The full definitions are in the bundled contract.

FAQ

What's inside

agent-eval-loop/
├── skills/
│   ├── start/         # the relai/ template and checkers
│   ├── discover/      # requirements, risks, open questions, trace triage
│   ├── trace-review/  # project-owned trace review apps and human feedback
│   ├── eval/          # harness, test design, hardening, mocks, benchmarks, running and scoring
│   ├── improve/       # regression checks, agent-chosen experiments, cost and latency
│   ├── loop/          # contract, new evidence, iterations and experiment rounds
│   └── coding-evals/  # agent-neutral tasks for coding agents: sources, checks and comparison
├── examples/          # example launcher, judge, triager and native-runner scripts
├── hooks/             # optional Claude Code and Codex hooks: guard frozen tests, run checkers after edits
├── tests/             # tests for the checkers and examples
├── DESIGN.md          # the design and the reasons behind it
└── README.md

Workflows that need managed state add a relai/ folder, holding only what they use: discover adds the wiki, eval the harness and tests, improve and loop their records; trace-review and direct fixes add none. Its contract is in relai/README.md (template: skills/start/template/README.md).

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for setup, validation, and review guidance, and follow our code of conduct. Report suspected vulnerabilities privately using SECURITY.md. Ask questions and share examples in Discussions.

Run the checker tests with:

python3 -m unittest discover -s tests

Tests that build images require a running Docker-compatible engine or Podman and are skipped without one. The reasoning behind each design choice is in DESIGN.md.