Skip to content
Projects

awslabs/AutoHelix has 35 GitHub stars and 5 forks. GitHot measured +1 stars in 24 hours. Last checked 2026-11-15 14:29 UTC. Last push 2026-11-13.

How these counts are measured

35 stars5 Forks +1 in 24h

What you can do

  • budget.time — caps total wall-clock for the whole run. Checked before each iteration starts; doesn't interrupt one in progress.
  • agent.timeout — per-iteration deadline, a hard kill. Where the backend supports it, a hook injects a countdown into the agent's context after each tool call, so it can wrap up and write notes before the deadline hits.

Measured star growth

in 24h
+1
in 7 days
Collecting history
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

Agent-powered iterative optimization: put an AI agent in a verified improvement loop.

AI

README.md

Original

AutoHelix

Put an AI agent in a verified improvement loop.

Set gates and what to observe; the agent proposes changes in a loop.
Only changes that pass the gates are accepted. Improvements compound across iterations.

works with Claude Code · Codex · OpenCode

Python 3.11+ License: Apache 2.0

How it works · Quick start · Examples · Configuration · Other features · FAQ

autohelix run — optimizing the sorting example Example autohelix.yaml configuration for the sorting task

AutoHelix is a lightweight harness that lets you point an agent at a goal and leave it looping unattended. It provides the isolation, validation, scope, and budget control to make that safe, and it's general enough for any measurable goal, from optimizing code to training models.

Looping an agent can be far more effective than a single prompt, because later iterations fix and build on what came before, and because it gives you a knob for controlling how much effort (time or cost) to put toward the goal rather than letting the agent decide when it's done. AutoHelix makes this pattern easy to run and customize, so you can let the agent climb for as long as you want.

How it works

AutoHelix iteration loop: copy → agent → checks → merge or discard → repeat

You define a goal, the gates a change must pass, and what to observe (benchmarks, reviews, artifacts). The agent proposes changes and the harness decides if they land.

Core features:

  • Isolation. Each iteration runs in fresh private copies of the code. A rejected attempt's code is cleanly discarded, and what was checked is exactly what gets accepted.
  • Scope enforcement. Only files marked editable can be changed; changes elsewhere are discarded, and the agent is told.
  • Gates decide. Tests, type-checks, any command that must exit 0, a yes/no question to a read-only agent (ask:), or a metric compared with the last accepted iteration. Fail a gate and the iteration is rejected; its change is kept aside for the next agent as a patch.
  • Observations inform. Benchmarks, logs, artifacts and LLM reviews are saved by name and shown to the next iteration; their numbers are the run's metrics.
  • Persistent memory. Each iteration is a fresh session, but the agent reads and writes notes that survive across all iterations, including rejected ones.
  • Budget control. Cap the run by iterations, time, or cost. Codex cost uses an API-equivalent estimate; see cost tracking.
  • Stop conditions. Optional stop: commands end the run when one exits 0, checked on the baseline and after accepted iterations. Use a script, a metric (jq -e '.missing == 0' "$AUTOHELIX_METRICS") or autohelix ask -- "Is the goal complete?"; see stop conditions.

Quick start

Install from source:

git clone https://github.com/awslabs/AutoHelix.git
cd AutoHelix
python -m venv .venv && source .venv/bin/activate
pip install -e .

Or pip install autohelix for the CLI and portable setup skill, without the repository's example projects.

AutoHelix uses Claude Code by default (looks for claude on your PATH). Codex and OpenCode are also supported — see agent setup.

To prepare the experiment with your existing coding agent, install the packaged skill into the target project:

autohelix skill install --agent codex --scope project --path my-project
# Use --agent claude or --agent opencode for those agents.

Your coding agent helps define the experiment. AutoHelix runs repeated attempts using the configured scope, gates, and observe entries.

  1. Prepare and review

    Set up AutoHelix to improve [goal]. [Optional instructions.]

    The agent shows the complete proposed config and walks through each section: what it controls, why it was chosen, and how to adjust it, including effective defaults. Review it before project files change. Discuss revisions; when you agree to save and commit, it writes, validates, and commits the setup, then gives the launch command.

  2. Run and manage

    Run and manage AutoHelix using the existing config. [Optional changes or budget limits.]

    The agent applies agreed changes to the configuration and supervises the run through termination.

Replace [goal] with the desired outcome and add any scope, evaluation, or budget preferences. You can also ask the agent to set up and manage the experiment in one request.

The complete skill and its references ship with pip/wheel installs; no AutoHelix checkout is needed. Installation is explicit, and existing skill folders are preserved unless you pass --force. See the conversational walkthroughs and skill installation and export.

Or start from the project itself:

cd my-project
autohelix setup

In a terminal it offers to start Claude Code, Codex or OpenCode with the bundled setup skill (nothing is installed), or writes the commented template autohelix.yaml. With an agent, review the proposal before it saves and commits the agreed setup. With the template, edit it using the configuration reference and commit the intended setup changes. Then launch with autohelix run. For several loops in one project, autohelix setup perf writes autohelix/perf.yaml, which autohelix run perf runs. Other commands are described in the CLI reference.

Examples

The examples/ directory includes concrete tasks ranging from review-driven refinement to workflow and algorithm optimization. To list what's available:

python examples/setup_example.py --list

Code optimization. AutoHelix can iteratively improve the performance of existing code while preserving user-defined correctness checks. We tested this capability on two established benchmarks: AlgoTune for CPU algorithms and KernelBench for GPU kernels. In both evaluations, median speedup increased with each iteration, reaching approximately 8.5× across 154 AlgoTune tasks and 3.12× across 270 KernelBench tasks after five iterations.

Median AlgoTune speedup over five AutoHelix iterations
AlgoTune · CPU algorithms
Median KernelBench speedup over five AutoHelix iterations
KernelBench · GPU kernels

Both runs used Claude Opus 4.8; KernelBench ran on an H200. The AlgoTune result uses our own evaluation setup, which is not identical to its official harness. See the examples/algotune and examples/kernelbench setups, plus the results/algotune and results/kernelbench methodology and per-task results.

Model training. The same loop works when each iteration is a training run rather than a code edit: the agent writes the training code and trains a model, and AutoHelix scores the result with a frozen eval the agent can't modify. In examples/posttrain (inspired by PostTrainBench), the agent is asked to post-train Qwen3-4B-Base on GSM8K. Given an open-ended goal (no method prescribed), the agent worked its way up from supervised fine-tuning through rejection sampling to reinforcement learning (GRPO) — reaching 93% accuracy over 8 iterations on one H200. See the ML Experiments Guide for setup tips.

GSM8K accuracy climbing from 41% to 93% over AutoHelix iterations

Advanced agent loops. AutoHelix can also support longer-running autonomous work. The research example iterates through experiments, evidence, and reports; task-queue works through a persistent list of coding tasks; and workflow-optimization improves a model-powered workflow's prompts, skills, context, and orchestration against a frozen evaluator.

In the wild. Beyond the bundled examples, we've used AutoHelix on real projects. In Hybrid Model Factory — an open-source toolkit for training hybrid architectures (Attention + SSMs) from our team — the agent autonomously developed a faster sequence-parallelism implementation. Over multiple runs (with some algorithmic hints in the goal), it arrived at a state-passing approach that propagates recurrent state between GPUs instead of redistributing the full sequence, reaching ~5× faster per layer and 1.6× faster end-to-end training at 128K sequence length on a 27B model.

Configuration

A run is defined by a single config file, autohelix.yaml (or autohelix/<name>.yaml, one per loop). Every field is documented in docs/config.md.

Annotated autohelix.yaml configuration

The same config as text
goal: |
  Optimize the data pipeline for throughput
  while maintaining correctness on all inputs.

gates:                       # anything that can reject an iteration
  - pytest tests/ -x
  - mypy src/ --strict
  - metric: throughput
    regress: 5%              # at most 5% worse than the last accepted iteration

observe:                     # feedback for the next agent, numbers for the harness
  - name: bench
    command: python benchmark.py   # prints ##autohelix[throughput=...]
    timeout: 30m
    metrics:
      throughput: higher
      latency_p99: lower
    capture: [results/profile.json]
  - name: review             # a read-only LLM review of the change
    review: What is the most important next fix?

scope:
  editable: [src/pipeline/, src/transforms/]

agent:
  backend: claude
  model: claude-opus-4-8
  timeout: 30m

budget:
  iterations: 10
  cost: 30
  time: 4h

Gates run first, in order; observe entries run next, and a failing one doesn't reject anything (unless a metric: gate needs its number). Configs from earlier versions (constraints:, metrics:, reviewer:, ...) are errors that name their replacement; see Old → new.

Project layout

my-project/
├── autohelix.yaml       # your config (or autohelix/<name>.yaml, one per loop)
└── .autohelix/          # run state (gitignored)
    ├── runs/<id>/       # one persistent state directory per run
    │   ├── run.json     # identity, starting commit, target branch
    │   ├── history.jsonl # one record per iteration: results and each model call's usage/log
    │   ├── notes/       # agent's persistent notes
    │   ├── hints.md     # messages from `autohelix hint`
    │   ├── report.md    # the run summary (and report.html, the run page)
    │   ├── iterations/NNN/ # one folder per iteration (000 is the baseline): record.json,
    │   │                #   prompt.md, agent.log, gates/<k>/, observe/<name>/, stop/<k>/
    │   ├── logs/        # run logs per invocation
    │   └── .internal/   # live status, usage, notes versions, superseded attempts
    └── .internal/       # selected run id, helper scripts and lock files

Temporary per-phase candidate copies live outside the project, in ${XDG_CACHE_HOME:-~/.cache}/autohelix/copies/ (set AUTOHELIX_COPIES_ROOT to move them).

Each autohelix run starts a new run; autohelix resume [<id>] [-n MORE] continues one, autohelix archive <id> sets one aside and autohelix delete <id> removes it. Use --land manual to keep validated commits on the run's refs/autohelix/<id> until autohelix land. By default, accepted changes land on the starting branch when a run segment (one invocation) ends. A resumed run can coexist with unrelated dirty checkout files. See CLI reference.

Other features

Report

When a run finishes, an agent reads the run's history, notes, logs and commits and writes a short summary to .autohelix/runs/<id>/report.md. autohelix report writes it on demand, e.g. after an interrupted run. See Configuration → Report.

Reviews and progress tracking

Beyond benchmarks, an observe: entry with review: <prompt> gets qualitative feedback from a read-only LLM on exactly the candidate (autohelix review, also usable by hand). Each review is saved with the iteration's observations and the next agent reads it. With previous: true and metrics: {missing: lower}, a review keeps its own status of a worklist across accepted iterations, and stop: can end the run when the number reaches zero. See Concepts → Reviews and Progress tracking.

Viewer and run page

autohelix view follows a run in the terminal: its status and iterations, with the agent's live output below; autohelix dashboard lists all runs. autohelix view --web serves a live page for a run on 127.0.0.1: status, metric chart, iterations with gates (and their output), notes, diffs, observations (reviews included), stop checks and agent activity, and the run summary once it is written. When a run stops, AutoHelix saves the same page as one file, .autohelix/runs/<id>/report.html, to open in a browser or share. See Concepts → Run page.

Time limits

Two time controls:

  • budget.time — caps total wall-clock for the whole run. Checked before each iteration starts; doesn't interrupt one in progress.
  • agent.timeout — per-iteration deadline, a hard kill. Where the backend supports it, a hook injects a countdown into the agent's context after each tool call, so it can wrap up and write notes before the deadline hits.

See Config → Agent and Budget.

Docker sandbox

Optionally run AutoHelix inside Docker for filesystem isolation. The agent can't reach your home directory, SSH keys, or other projects. Build the image with docker/build.sh and run via docker/run.sh. See Docker sandbox.

Hints

autohelix hint "..." (run in a second terminal) drops a note the running agent picks up at the start of its next iteration. Useful to steer a run without stopping it. See CLI → hint.

Full customization

Beyond the config file, you can replace the full per-iteration prompt template (prompt_template:, with a documented set of variables) and control how the agent organizes its persistent notes. See Concepts → Prompt and Concepts → Notes.

Several runs at once

Runs started with autohelix run --group NAME run side by side in one project and see each other's notes, history and accepted code at the start of every iteration. Compare them on autohelix dashboard, autohelix land the winner and autohelix archive the rest. See CLI → Groups.

FAQ

How does AutoHelix differ from other "agent in a loop" tools?

Running AI agents iteratively is becoming a popular strategy — a kind of test-time scaling for agents, where more iterations buy more quality. Tools like autoresearch, Ralph, and Claude Code's /goal and /loop commands each implement variations of this pattern.

Compared to other tools, AutoHelix adds a minimal amount of structure to ensure that long-running autonomous loops don't go off track, so you can let your agent work while keeping control over the direction.

AutoHelix autoresearch Ralph /goal /loop
Goal-directed ✓ any metric(s) ✓ val loss ✓ task list ✓ boolean ✗
Agent-agnostic ✓ ✓ ✓ ✗ ✗
Validation gates ✓ ~ self-checks ✗ ✗ ✗
Isolation on failure ✓ private copy ~ self-resets ✗ ✗ ✗
Fresh context each iteration ✓ ✗ ✓ ✗ ✗
Memory across iterations ✓ notes, metrics, artifacts session progress file session session
Scope enforcement ✓ ~ by instruction ✗ ✗ ✗
Budget control ✓ iters, time, cost ✗ iters only iters only ✗
Time-aware agent ✓ ✗ ✗ ✗ ✗
Parallel exploration ✓ ✗ ✗ ✗ ✗
What about reward hacking?

The agent optimizes what you measure: if your benchmark has loopholes, the agent may find and exploit them, and more iterations apply more pressure to do so. Autonomous loops shift the burden from "write perfect code" to "write strong checks."

You can look out for gaming by reading autohelix report, the agent's notes, or a review's feedback. Then close the loophole with a new gate and re-run.

Is it safe to run unattended?

AutoHelix isolates the agent at two levels:

  1. Candidate copies (default) — each phase of an iteration runs in a separate private copy, outside your project, with its own git metadata. Rejected iterations never reach your branch (their trees stay on hidden refs until the run is deleted); accepted ones are recorded on the run ref and landed separately.
  2. Docker sandbox (optional) — filesystem isolation: the agent can't access your home directory, SSH keys, or other projects. (Network is not isolated, since the agent needs it to reach the model API.)

Copy isolation protects your codebase from bad iterations, but it is not a security sandbox — the agent still runs on your host and could in principle access the filesystem or network. For focused optimization tasks (the typical use case), the agent works toward a narrow goal, not exploring adversarially, so running without a sandbox is usually a reasonable tradeoff. For stronger guarantees, or on sensitive machines, use the Docker mode.

Contributing & feedback

We'd love to hear how you're using AutoHelix — the goals you point it at, patterns you wish it supported, or rough edges you hit. Please open an issue for bugs, ideas, or use cases you'd like to share. Contributions are welcome.

Citation

@software{autohelix2026,
  title   = {AutoHelix: Verified Iteration for AI Agents},
  author  = {Trager, Matthew and Mansimov, Elman and Zhang, Yi and Xia, Wei and Soatto, Stefano},
  year    = {2026},
  version = {0.1.1},
  url     = {https://github.com/awslabs/AutoHelix}
}

License

AutoHelix is licensed under the Apache License 2.0.