Skip to content
Projects

ChaoYue0307/awesome-loop-engineering

View on GitHub

ChaoYue0307/awesome-loop-engineering has 60 GitHub stars and 9 forks. GitHot measured +1 stars in 24 hours. Last checked 2026-11-16 19:43 UTC. Last push 2026-11-03.

How these counts are measured

60 stars9 Forks +1 in 24h

Measured star growth

in 24h
+1
in 7 days
Collecting history
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

🔁 Build reliable recurring AI-agent systems: 1022 resources, 22 operational patterns, 22 loop contracts, 8 runtime starters, an interactive atlas, and a structured dataset.

Learning

README.md

Original

Visit the Awesome Loop Engineering website

Awesome Loop Engineering

A practical field guide and implementation kit for recurring, stateful, verified AI-agent systems.

Awesome Loop Engineering: the Prompt, Context, Harness, and Loop layers

Awesome project site Hugging Face dataset Quality checks resources patterns schema-checked contracts runtime starters license PRs welcome

📦 Build from this | 🧩 Pattern library | ▶️ Examples | 🧾 Schema | 🌐 Website | 🤗 Dataset

🌍 Languages: English | 中文 | Español | Français | Deutsch | 日本語 | 한국어 | Português | Help translate

⭐ Star to find it again · 🔀 Fork the contracts and dataset schema · 🔔 Watch Releases and Discussions

Loop Engineering is the operating layer for recurring AI-agent work. It defines how work enters, what an agent may do, which external evidence proves completion, what state survives, and whether the system retries, reports, escalates, or exits.

This field guide connects 1022 papers, docs, tools, benchmarks, and guides to 22 operational patterns, 22 schema-checked contracts, and 8 runtime starters, so a recurring job can move from evidence to a reviewable implementation.

The reliability gap: prompts, context, and harnesses can improve one run; recurring work also needs explicit triggers, verification, durable state, bounded budgets, and human handoff. Software event loops, control theory, growth loops, generic automation, and one-off prompting are outside this map.

Start In 60 Seconds

🧭 Explore the field Open the Resource Atlas Filter 1022 sources by goal, loop layer, lifecycle stage, artifact type, and evidence.
▶️ Build one loop Follow a worked path Move from a recurring job to its pattern, contract, runtime, and completion signal.
📊 Analyze the map Load the dataset Query 50 publication, evidence, lifecycle, source, and repository fields in CSV, JSONL, or Parquet.
🔔 Follow useful updates Read the current release · Follow the digest Use Watch → Custom → Releases and Discussions for a low-noise update feed.

Contents

Implementation Kit

Move from evidence to implementation through four connected layers:

Start with What it gives you Use it to Example
📚 1022 resources Papers, official docs, tools, benchmarks, and critiques with publication and evidence fields Understand the design space and open the original work behind a claim Compare verification methods in the Resource Atlas
🧩 22 operational patterns Symptom-first playbooks with roles, gates, state, budgets, escalation, and worked scenarios Choose how a recurring job should operate "CI keeps failing" becomes a CI repair loop
🧾 22 schema-checked contracts One schema-valid JSON specification for every pattern Make permissions, evidence, limits, and human handoff reviewable Adapt the CI repair contract to your repository
▶️ 8 runtime starters 3 dependency-light executables plus 5 copy/paste runtime templates Wire a contract to a session, schedule, CI event, or durable worker Start with test repair, threshold monitoring, or queue processing

Fastest build path: name the recurring symptom, inspect its pattern, adapt the linked contract, then choose a runtime for the required persistence, isolation, and permissions.

Browse the interactive Resource Atlas, query the structured dataset, or use one of 8 language entry points.

See it end to end: the four worked paths cover CI repair, knowledge-base refresh, bounded queue processing, and read-only threshold monitoring, including commands and verifiable completion signals.

Two-minute dry run: clone the repository and inspect a bounded queue item without an agent account or extra dependency.

git clone https://github.com/ChaoYue0307/awesome-loop-engineering.git
cd awesome-loop-engineering
printf '%s\n' '{"id":"demo","objective":"Validate one queue item"}' > /tmp/loop-queue.jsonl
python3 examples/runnable/queue-worker-loop.py --queue /tmp/loop-queue.jsonl --dry-run

Mental Model

Prompt engineering asks: what should I say to the model?

Context engineering asks: what state and knowledge should the model see?

Harness engineering asks: what tools, permissions, tests, sandboxes, and feedback should surround the agent?

Loop engineering asks: what recurring system should discover work, delegate to agents, verify results, persist state, decide next actions, and re-run when the human is no longer in the inner loop?

Prompt, context, and harness engineering make one agent run better. Loop Engineering makes agent work repeatable, observable, and governable over time.

Diagram of the engineering stack: Prompt, Context, and Harness Engineering improve one run; Loop Engineering governs recurring agent work over time
Figure 1. Prompt, context, and harness engineering improve one run; Loop Engineering governs recurring work over time.

Loop shape:

Objective
  -> Trigger / cadence
  -> Discover / intake work
  -> Delegate to agents
  -> Act in an isolated workspace
  -> Verify with tests, evals, traces, or reviewers
       -> if failed: feed back the evidence and retry
       -> if passed: persist state and decide what happens next
  -> Repeat, report, open a PR, or escalate to a human

Loop Engineering lifecycle: Intake, Delegate, Act, Verify, Persist, Decide; Decide retries by feeding evidence back, escalates to a human, or exits when the goal is met
Figure 2. Evidence moves through a recurring, stateful, verified loop toward retry, escalation, or exit.

Reading Paths

Choose a goal before opening the full catalog. Learn the field through the definition and research foundations; build through a pattern, contract, and runtime; or review reliability through verification, state, security, benchmarks, and critiques.

Goal Start here Continue with Best for
🧭 Learn Working definition Start Here → Research Foundations Newcomers and researchers
🧠 Study model recurrence Model-Level Recurrence Research Foundations → Benchmarks And Evaluation Model, reasoning, and agent researchers
✏️ Design Loop Contract Pattern Library → Core Loop Primitives → Agent Workflow Patterns Architects and loop designers
🛠️ Build Runtime selection Official Runtime Guides → Coding-Agent Loop Systems → Orchestration Agent and platform builders
💾 Persist State and context Core Loop Primitives → Operations Playbooks Long-running and resumable loops
✅ Verify Verification Gates Benchmarks And Evaluation → Critiques And Limitations Eval and reliability engineers
🛡️ Govern Securing Unattended Loops Operations Playbooks → Critiques And Limitations Operators, approvers, and security
📦 Apply Templates And Patterns Examples And Schema → Community Gallery → Contributing Teams adopting or sharing loops

For analysis, the Hugging Face dataset exposes collection, lifecycle, audience, evidence, source-status, publication, and repository-stat facets for every resource.

Fastest conceptual path: read the Working Definition, inspect the Loop Contract, choose a symptom in the Pattern Library, and follow a worked implementation.

Choose Your Loop

Choosing a loop means choosing the operating policy around an agent, not the model itself. A pull-request loop, deploy loop, and adversarial-testing loop may use the same model, but they need different triggers, permissions, evidence gates, durable state, retry budgets, and human owners. Those controls decide what the agent may do, what counts as success, what the next run can remember, and when automation must stop.

A generic "keep trying" loop collapses those differences. It can retry a judgment call, let the acting model approve its own work, lose evidence between runs, or change a high-risk system when it should only observe and escalate. A named pattern supplies a reviewable starting policy for a recurring job; its contract makes that policy exact for your environment.

First Decide Whether The Work Should Loop

Use a loop when... Prefer one supervised run when...
Work returns through a schedule, event, queue, or recurring condition. The task is genuinely one-off.
An external check can prove progress or completion. "Done" depends mainly on open-ended human judgment.
Saved evidence, checkpoints, or receipts improve the next run. No useful state should survive the session.
Permissions, retries, time, and cost can be bounded in advance. A safe permission or budget boundary cannot be stated.
A named human owns ambiguity, exceptions, and high-impact decisions. Every meaningful step already needs live human direction.

Choose In Four Decisions

Decision Ask What it fixes
1. Name the recurring job What signal keeps returning, and what outcome should improve? Objective, trigger, and intake
2. Define proof What evidence can a reviewer or machine inspect to decide "pass"? Verification gate and exit condition
3. Bound action What may the agent read or change, and what must it never do? Workspace, permissions, non-goals, and budget
4. Plan the next pass What should persist, retry, escalate, or stop after each result? Durable state, handoff, and next action

Select the pattern for the class of work, adapt its contract to your exact controls, then choose the runtime based on persistence, isolation, and permissions. Start from the problem and its verified finish below, or compare every field in the pattern matrix.

When you say... Start with The loop can finish when...
"My PR is stuck" PR babysitter Required checks pass, review threads are resolved, and merge state is current.
"CI keeps failing" CI repair loop The original failing command passes with a scoped patch.
"The docs may be stale" Docs drift collector Verified code/docs mismatches are patched and examples still run.
"A deploy needs monitoring" Deploy verifier Synthetic checks and rollout thresholds remain within policy.
"Feedback is noisy" Feedback clusterer Themes cite source items and separate frequency from severity.
"Dependency updates pile up" Dependency triage loop Safe updates pass tests and risky upgrades have an owner-backed escalation.
"Agent evals regressed" Evaluation regression loop Targeted evals return to the accepted baseline without scorer changes.
"Sensitive changes need review" Security review loop Findings cite concrete evidence and approval boundaries stay intact.
"Agent spend is rising" Cost-control loop Spend falls on a comparable workload without quality regression.
"I need recurring bug discovery" Bug hunting loop Every accepted finding has reproducible steps or a failing test.
"A change needs sign-off" Enterprise approval loop Every required gate has a recorded decision and audit trail.
"An incident just paged" Incident response loop Impact, evidence, timeline, and accountable owner are recorded.
"A dataset keeps drifting" Data-quality loop Hard quality rules pass before a new version is promoted.
"Release notes are a chore" Release-note loop Every shipped change maps to a merged source and audience-facing note.
"Model choice is ad hoc" Model-routing loop Routing meets quality, latency, privacy, and cost tolerances.
"A stable system should improve" Benchmark optimization loop Repeated measurements confirm improvement with protected metrics intact.
"The agent's knowledge is stale" Knowledge freshness loop A versioned corpus passes provenance, freshness, retrieval, and leakage gates.
"Runs repeat mistakes they already made" Agent memory lifecycle loop Governed memory records pass provenance, scope, and recall gates between runs.
"Parallel agents collide on one repo" Fleet coordination loop Isolated workers land serially through a verified merge queue.
"Performance regressed" Performance regression loop A controlled benchmark confirms recovery with correctness intact.
"A UI accessibility check failed" Accessibility regression loop The exact regression is fixed and required human criteria are approved.
"The agent needs adversarial testing" Adversarial red-team loop Findings are reproduced, minimized, privately reported, and regression-tested.

After choosing the pattern, compare runtime persistence, isolation, permissions, and escalation.

Working Definition

Loop Engineering is the AI and coding-agent practice of designing recurring systems that discover work, delegate it to agents, verify results, persist state, decide next actions, and run again on a cadence, event, or until a verifiable goal is reached.

Concept Guides

These vendor-neutral guides define the concept, boundaries, vocabulary, and reviewable artifacts.

Resource Published at Contribution Evidence
🧾 Working Definition 2026 · GitHub
Project documentation
Short definition, positioning, minimal loop test, and citation note. Project artifact
🧾 Loop Engineering Manifesto 2026 · GitHub
Project documentation
Concise statement of the concept, commitments, non-goals, and success standard. Project artifact
🧾 Loop Engineering Taxonomy 2026 · GitHub
Project documentation
Classification by trigger, intake, verification, state model, topology, and operating domain. Project artifact
⚠️ Loop Engineering Anti-Patterns 2026 · GitHub
Project documentation
Common failure modes such as prompt loops with no contract, infinite retries, model self-approval, hidden state, and unsafe autonomy. Project artifact
🧾 Comparison Guide 2026 · GitHub
Project documentation
Distinguishes Loop Engineering from prompt engineering, context engineering, harness engineering, workflow automation, agent workflows, and evaluation loops. Project artifact
🧾 Sourced Signals And Quotes 2026 · GitHub
Project documentation
Short sourced signals from linked public materials that anchor the emerging concept. Project artifact
🧾 Outreach Kit 2026 · GitHub
Project operations guide
Conservative messages for inviting corrections, sources, and real-world loop patterns. Project artifact

How To Judge Each Source

Open the original work before relying on a summary. Use the contribution, novelty, impact, and evidence fields to compare approaches and understand what supports each entry.

Dimension Evidence standard
Scope Every entry must directly inform recurring AI-agent systems, offer concrete value to builders or researchers, and link to evidence that readers can inspect.
Link availability At the latest source check on 2026-11-03 UTC, 963 public links opened successfully, 8 required access, 51 pointed to files in this repository, and none were broken or unreachable.
Evidence label Labels distinguish published research, official documentation, inspectable implementations, practitioner analysis, risk analysis, and discovery indexes. They describe the kind of source behind an entry, not its scientific quality.
How summaries are written Summaries explain each work's contribution in original language rather than copying abstracts. They do not imply author endorsement; follow the link to inspect methods, results, limitations, and current documentation.
Corrections Report an error through the annotation-correction form or a pull request. Corrections to contribution, novelty, impact, publication metadata, and source links take priority.
Licensing Original summaries, metadata, templates, and project documentation are CC0-1.0. Linked works retain their own licenses and terms.

Resource Type Legend

Type Rows Includes
📄 Paper 500 Academic paper, preprint, or technical report
📝 Blog 117 Essay, field note, article, or practitioner write-up
📚 Docs 85 Official product, API, SDK, or platform documentation
🧰 Tool 146 Repository, framework, SDK, runtime, or implementation
🧪 Benchmark 80 Benchmark, eval suite, leaderboard, or evaluation dataset
🔁 Pattern 37 Operational playbook or reusable workflow
🧾 Template 25 Template, checklist, schema, guide, or contribution artifact
🧭 List 19 Adjacent directory, ecosystem map, or reading list
⚠️ Critique 13 Risk analysis, limitation, caveat, or skeptical take

Every row shows the work, its type as an icon defined in the legend above, its original publishing platform or venue, the contribution that matters for Loop Engineering, and a plain-language evidence label. Author lists are shortened for scanning; the queryable dataset exports and Resource Atlas retain full authorship, dates, venues, identifiers, contribution, novelty, impact, evidence signal, link status, and repository statistics.

Evidence labels describe the kind of source behind an entry, not its scientific quality. Each label means the same thing on every row, so it is defined once here rather than repeated in the tables; a row carries an extra note only when its link was access-restricted or unavailable at the latest check.

Evidence label Rows What it means
Research paper 62 Published or accepted research record
Research preprint 442 Preprint; inspect methods and evaluation
Official documentation 74 Primary product or standard behavior
Technical documentation 11 Technical reference from the source
Source implementation 130 Inspectable source and runtime behavior
Implementation 12 Working implementation or runtime
Benchmark or evaluation 80 Repeatable tasks, scores, or evaluation data
Project artifact 50 Schema, example, guide, or repository documentation
Reusable artifact 2 Adaptable template, schema, or guide
Operational pattern 11 Transferable operating practice
Practitioner analysis 117 Experience-backed implementation context
Risk analysis 12 Failure modes, limits, or adoption cautions
Discovery index 19 Broader ecosystem coverage for finding related work

‡ The linked source required access at the latest check.

Start Here

Direct resources about the new AI/coding-agent meaning of Loop Engineering.

Resource Published at Contribution Evidence
📝 Loop Engineering by Addy Osmani 2026 · AddyOsmani.com
Addy Osmani
Addy Osmani's framing of loop engineering as the layer above manually prompting coding agents, with concrete primitives across Codex and Claude Code; also on Substack with the original discussion trail and Steinberger and Cherny quotations. Practitioner analysis
📝 Peter Steinberger on designing loops 2026 · X (formerly Twitter) The June 2026 post - "you shouldn't be prompting coding agents anymore, you should be designing loops that prompt your agents" - that catalyzed the current discussion. Practitioner analysis
📝 Boris Cherny: five tips for running Opus autonomously for hours or days 2026 · X (formerly Twitter) The Claude Code creator's compact loop recipe: auto-mode permissions, dynamic workflows, /goal or /loop, the cloud runner, and end-to-end self-verification. Practitioner analysis
📝 Loop Engineering by Cobus Greyling Substack
Cobus Greyling
Concise explanation of the shift from prompting agents to designing loops that discover work, delegate, verify, persist, and continue. Practitioner analysis
📝 Stop Prompting. Design the Loop. 2026 · pulumi
Engin Diri
Practical breakdown of loop building blocks - automations, worktrees, skills, connectors, subagents - plus external memory and verification through oracles such as tests and builds. Practitioner analysis
📝 Writing Loops, Not Prompts, Explained rico.codes Rico Kahler's break-even model for when a recurring task justifies building a loop instead of prompting, with stop conditions, evidence collection, and an execution-horizon framing for moving from execution-bound to judgment-bound work. Practitioner analysis
📝 Loop Engineering: A Guide for Engineers and Practitioners Medium Adnan Masood's practitioner guide that organizes loop design into triggers, topologies, verifiers, and termination rules, with coverage of failure modes, cost control, and observability for production agent loops. Practitioner analysis ‡
📝 Loop Engineering: When Generation Gets Cheap, Judgment Gets Expensive Substack
Stephane Derosiaux
Stephane Derosiaux's essay on the economics of the loop layer (generation becomes abundant while judgment becomes the bottleneck), proposing evaluator agents that must act rather than merely review, and cataloging failure modes such as unverified merges and quota depletion. Practitioner analysis
📝 Andrew Ng on Loop Engineering and the Three Loops of AI-Native Product Development 2026 · X (formerly Twitter) Andrew Ng's letter laying out three product-development loops (agentic coding in minutes, developer feedback in hours, external feedback in days) and arguing that human-in-the-loop persists wherever the human knows something the AI does not. Practitioner analysis
📝 From Prompting Agents to Loop Engineering 2026 · X (formerly Twitter) DAIR.AI founder Elvis Saravia's X article examining the claim that you should stop prompting coding agents and start designing loops that prompt them for you. Practitioner analysis
📝 My Lord! AI Programming Undergoes Another Major Shift eu.36kr.com Broad coverage of the Boris Cherny and Peter Steinberger discussion, including the distinction between cold-start scripts and persistent agent loops. Practitioner analysis
📝 The Anthropic leader who built Claude Code ditched prompting - now he writes loops 2026 · The New Stack
Janakiram MSV
The New Stack's report on Boris Cherny's shift from prompting to loop writing and what it changes about developer workflow. Practitioner analysis
📝 Engineering for Agents That Never Sleep Substack
Nader Dabit
Cognition's Nader Dabit predicts the human-initiated share of Devin sessions will invert from 70/30 to 10/90 within a year as signals like alerts and failing tests trigger agents directly, recasting the engineer's job as designing triggers, constraints, and quality gates. Practitioner analysis
📝 Loop Engineering Orange Book 2026 · GitHub
alchaincyf/loop-engineering-orange-book
Plain-language bilingual (Chinese and English) field guide to loop engineering by HuaShu, framing the discipline as one floor above harness engineering: the outer system that decides when and why agents run. Practitioner analysis
📝 How I AI: How to Write AI Agent Loops in Claude Code and Codex lennysnewsletter.com
Lenny Rachitsky
Mozilla distinguished engineer Brian Grinstead demonstrates goal-based and scheduled loops, including a daily PR-review loop with per-PR subagents, on Lenny's Newsletter. Practitioner analysis
📄 Proof-or-Stop: Don't Trust the Agent, Trust the Evidence -- Loop Engineering for Verifiable Evidence-Gated Lifecycle Control 2026 · arXiv
Jek Huang et al.
Defines evidence-gated lifecycle control for agent loops and reports zero false-DONE outcomes across 10 scenarios and zero accepts across 18 tampering classes; its 9,240-cell ablation identifies which gates prevent error amplification, while noting the evaluation covers one model family and 24 tasks. Research preprint

What Counts As Loop Engineering

Qualifies Does not qualify
AI/coding-agent loops that coordinate prompts, context, harnesses, verification, and state over repeated agent runs Software event loops, UI/game loops, or control theory loops
Scheduled, goal-driven, or event-triggered agent work Generic cron jobs with no agentic reasoning or verification
Agent loops with durable state, worktrees, checkpoints, traces, or progress files One-off prompt examples with no loop, state, or feedback signal
Verification loops using tests, CI, evals, reviewers, or deterministic gates Pure AI news, generic product pages, or marketing copy
Multi-agent maker/checker/delegation patterns Broad agent lists without specific loop-design relevance

Model-level recurrence belongs in the map when it explains the model inside an agent: a learned block or latent state is iterated to allocate compute, refine reasoning, or simulate a world. It is an adjacent foundation, not a complete operational loop. The outer system still needs work intake, tools, external verification, durable state, budgets, escalation, and exit.

The Loop Contract

A Loop Contract is a reviewable operating specification for one recurring agent job. It is not a legal agreement, a long prompt, or a particular runtime. It records the decisions a person normally supplies during a one-off session - what work is authorized, what the agent may touch, what evidence counts, what survives, and when control returns to a human - so those decisions remain stable across runs.

The contract becomes necessary when an agent runs from a schedule, event, queue, or goal instead of waiting for live supervision. Unanswered questions become hidden defaults: the agent may select the wrong work, widen its own scope, approve its own output, forget a previous failure, or retry without a stopping rule. Making the policy explicit lets builders, reviewers, security teams, and operators inspect the same boundaries before the loop acts.

Prompt, context, and harness choices improve one run. A pattern describes how a class of recurring work should operate. The contract fixes the exact policy for one implementation, and the runtime executes it.

Loop Contract cards: objective, trigger, intake, workspace, context, delegation, verification, state, budget, escalation, and exit
Figure 3. The Loop Contract makes recurring agent work explicit, reviewable, and bounded.

Read It As Three Decisions

Phase Contract parts What the phase decides
1. Set up Objective, trigger, intake, workspace Why a run is authorized, which work qualifies, and where action is allowed.
2. Run Context, delegation, verification, state What the agents know, who acts and checks, what proves progress, and what is remembered.
3. Govern and close Budget, escalation, exit, next action How autonomy is capped and whether the loop retries, reports, hands off, or stops.
Part Decision the contract must make Common artifact
Objective Which measurable outcome is worth repeating toward? Goal, issue, PRD, runbook
Trigger Which event, schedule, or goal authorizes a run? Schedule, webhook, /loop, /goal, automation
Discover / Intake Which items qualify, and how are duplicates or stale work rejected? GitHub queries, Linear filters, CI failures, feedback stream
Workspace What may the agent read or change, and where is execution isolated? Worktree, sandbox, branch, container
Context Which instructions and current evidence must be loaded fresh? AGENTS.md, CLAUDE.md, SKILL.md, docs, logs
Delegation Who explores, acts, and checks, and can the acting agent approve itself? Explorer, implementer, reviewer, judge
Verification Which external evidence must pass before progress or completion is valid? Tests, typecheck, lint, evals, trace graders
State Which checkpoints and receipts survive, and when are they updated? Progress file, database checkpoint, trace, issue comment
Budget How much time, retry, token, cost, or concurrency may one run consume? Max turns, max retries, token budget, time box
Escalation Which conditions require a named human owner or review channel? PR, issue, Slack alert, triage inbox
Exit What proves success, and what stops the loop without success? Acceptance criteria, passing checks, explicit blocked state

Why The Boundaries Matter

If the contract omits... The recurring failure is predictable
Trigger and intake The loop invents work, reprocesses the same item, or runs at the wrong time.
Workspace and permissions A narrow repair expands into unrelated files, credentials, or production changes.
Independent verification The acting model can declare its own output complete without external evidence.
Durable state and receipts The next run repeats failed attempts and cannot explain what already happened.
Budget, escalation, and exit The loop retries indefinitely, spends past its value, or stalls without an owner.

Worked Example: PR Babysitter

Suppose a team wants an agent to keep one pull request moving without asking a person to poll CI and review threads all day. "Keep this PR moving" is only a goal; the contract turns it into bounded operation.

Contract decision PR babysitter implementation
Authorize the run Run every two hours during working hours and after requested changes or failed required checks. Select only explicit blockers on one pull request.
Bound the action Use a dedicated branch or worktree. Allow narrow fixes, checks, and progress comments; prohibit force pushes, dependency upgrades, secrets access, and production changes.
Load and delegate Load AGENTS.md, CONTRIBUTING.md, the latest PR SHA, review threads, and CI logs. An explorer identifies the blocker, an implementer patches it, and a reviewer checks scope.
Prove and remember Require GitHub checks to pass, review threads to resolve, and the diff to stay scoped. Persist commands, check URLs, changed files, blockers, and the next action.
Cap and hand off Stop after three retries or 60 minutes. Escalate architectural decisions, repeated failures, reviewer disagreement, or any force-push requirement to the human owner.
Exit Succeed when the PR is merge-ready or waiting only on human review; stop without success when blocked, out of budget, or facing a judgment call.

Open the full pr-babysitter-loop.json, render it with python3 scripts/preview_loop_contract.py, or browse the 22 schema-checked contracts below.

Loop Design Checklist

Check Question
Name one objective Does the loop optimize for a specific outcome instead of a vague goal such as "improve the repo"?
Define the intake Where does work enter: PR comments, CI failures, issues, logs, eval failures, feedback, or schedule?
Isolate execution Does the agent act in a worktree, sandbox, branch, container, or read-only mode?
Write the feedback signal first Do tests, typechecks, lint, evals, policy checks, or trace graders exist before retries begin?
Persist state outside the model Does progress survive in files, issue comments, checkpoints, traces, or a database?
Separate maker and checker Does something other than the acting agent decide whether the work is done?
Put a budget on autonomy Are runtime, turns, retries, token spend, and concurrent workers capped?
Design escalation Is it clear when the loop should open a PR, file an issue, ask a human, or stop?
Keep receipts Are commands, evidence, changed files, and stop reasons recorded?

Loop Maturity Model

Loop Maturity Model progressing from manual prompting through scripted retry, scheduled, stateful, self-verifying, multi-agent, and production-supervised loops

The Loop Maturity Model is a capability ladder for one recurring agent workflow. It helps you choose the smallest operating model that can perform a job reliably. The level describes how the workflow is triggered, remembered, verified, divided, and supervised; it does not score model intelligence, team sophistication, or product quality.

How To Use The Model

Start with the job, not the target level. Identify the recurring outcome, evidence, risk, and human owner before choosing an architecture.

Treat levels as cumulative capabilities. A Level 4 loop still needs a bounded trigger and durable state; adding a verifier does not repair missing foundations.

Move up only for a recurring failure or operating need. Every level adds code, observability, permissions, and maintenance. Stop at the lowest level that meets the job's continuity, evidence, and blast-radius requirements.

Earn autonomy in order. Persist state before increasing unattended runtime, establish external verification before adding more agents, and add production controls before actions can affect users or infrastructure.

Level Operating model Why it exists Concrete example Progress signal
0 · Manual prompting A person holds the state, supplies each next instruction, and judges the result. Keeps ambiguous, one-off, or high-judgment work under direct supervision. A developer asks an agent to fix one failing test, reviews the diff, and decides whether to continue. The same task and feedback sequence recur often enough to encode.
1 · Scripted retry A bounded wrapper reruns one agent and feeds back an external failure. Removes repetitive reprompting while keeping one objective and one feedback signal. Run pytest; return the failing output to the agent for at most three repair attempts. The job should start from a schedule or event without a person launching it.
2 · Scheduled loop A schedule or event triggers fresh intake, one bounded run, and a report or artifact. Removes manual launch and makes cadence, idempotence, and no-work exits explicit. A nightly docs-drift scan opens a report only when code and documentation disagree. Work spans runs, repeats items, or loses context between invocations.
3 · Stateful loop Durable checkpoints and receipts record completed work, blockers, evidence, and the next action outside the model. Survives restarts, avoids duplicate work, and makes each run resumable and inspectable. A feedback clusterer stores processed IDs, prior themes, and the last successful checkpoint. Stable external evidence can decide whether a transition or completion is valid.
4 · Self-verifying loop Tests, evals, policy checks, traces, or an independent evaluator gate progress and exit. Replaces the agent's claim of success with reviewable evidence. A CI repair loop exits only when the original failing command passes and the diff remains in scope. One role becomes a bottleneck, duties must be separated, or independent review is required.
5 · Multi-agent loop Specialists divide discovery, action, review, and judgment through explicit handoffs and shared state. Adds separation of duties and parallel expertise without letting the acting agent approve itself. A security loop uses an explorer to find a candidate, a reproducer to confirm it, and a reviewer to judge severity and scope. The loop can affect production, customers, money, credentials, or other high-impact systems.
6 · Production-supervised loop Telemetry, least privilege, budgets, approvals, rollback, incident ownership, and human escalation govern every run. Controls blast radius and makes unattended production work observable, interruptible, and accountable. A deploy verifier watches rollout metrics, pauses on a breached threshold, preserves evidence, and hands rollback approval to the release owner. Harden service objectives, replay, permission reviews, failure drills, and cost controls as the workflow evolves.

Levels describe capability, not ambition. Many useful workflows should stop at Level 2 or 3. A dependable Level 3 loop with durable state is more valuable than a Level 5 design with vague goals, weak checks, or ceremonial agent roles.

Pattern Library

Practical loop patterns translate the abstract contract into runnable operating models. Each pattern documents the trigger, discover/intake step, agents, workspace, state, verification gates, retry budget, escalation path, and loop instruction.

Resource Published at Contribution Evidence
🔁 PR babysitter 2026 · GitHub
Pattern library
Repeatedly checks review comments, CI, merge conflicts, stale threads, and readiness to merge. Project artifact
🔁 CI repair loop 2026 · GitHub
Pattern library
Reproduces failing checks, patches narrowly, reruns evidence, and escalates when failures are outside scope. Project artifact
🔁 Docs drift collector 2026 · GitHub
Pattern library
Finds mismatches between docs and code, proposes small patches, and verifies examples. Project artifact
🔁 Deploy verifier 2026 · GitHub
Pattern library
Watches rollout signals, compares them with release expectations, and stops on anomalies. Project artifact
🔁 Feedback clusterer 2026 · GitHub
Pattern library
Periodically groups GitHub, Linear, Slack, support, or social feedback into actionable themes. Project artifact
🔁 Dependency triage loop 2026 · GitHub
Pattern library
Classifies dependency updates, applies safe groups, verifies them, and escalates risky upgrades. Project artifact
🔁 Evaluation regression loop 2026 · GitHub
Pattern library
Investigates degraded agent evals with baseline traces, targeted reruns, and repair proposals. Project artifact
🔁 Benchmark optimization loop 2026 · GitHub
Pattern library
Runs bounded experiments against a frozen benchmark and accepts only reproducible gains with correctness intact. Project artifact
🔁 Security review loop 2026 · GitHub
Pattern library
Reviews sensitive diffs with evidence-backed findings, safe permissions, and human approval boundaries. Project artifact
🔁 Adversarial red-team loop 2026 · GitHub
Pattern library
Discovers agent failures inside an authorized sandbox, then independently reproduces, minimizes, and reports them. Project artifact
🔁 Accessibility regression loop 2026 · GitHub
Pattern library
Repairs reproducible accessibility regressions while preserving required human review for non-automatable criteria. Project artifact
🔁 Cost-control loop 2026 · GitHub
Pattern library
Monitors agent workflow spend, identifies waste, proposes scoped savings, and preserves quality gates. Project artifact
🔁 Performance regression loop 2026 · GitHub
Pattern library
Profiles a measured regression and verifies a narrow fix against the same controlled workload and correctness gates. Project artifact
🔁 Bug hunting loop 2026 · GitHub
Pattern library
Discovers, reproduces, minimizes, and reports bugs with concrete evidence. Project artifact
🔁 Enterprise approval loop 2026 · GitHub
Pattern library
Drives a permissioned change through required gates and approvers with a full audit trail. Project artifact
🔁 Incident response loop 2026 · GitHub
Pattern library
Triages an alert into an owned, evidence-backed incident with a postmortem seed. Project artifact
🔁 Data-quality loop 2026 · GitHub
Pattern library
Validates each dataset refresh against quality rules and quarantines bad versions. Project artifact
🔁 Knowledge freshness loop 2026 · GitHub
Pattern library
Rebuilds a versioned retrieval corpus and promotes it only after provenance, freshness, quality, and leakage gates pass. Project artifact
🔁 Agent memory lifecycle loop 2026 · GitHub
Pattern library
Carries governed memory between recurring runs: explicit writes, scoped recall, audits, consolidation, and logged, reversible deletion. Project artifact
🔁 Fleet coordination loop 2026 · GitHub
Pattern library
Lands many parallel agent changes on one codebase through claim ledgers, isolated workspaces, and a verified serial merge queue. Project artifact
🔁 Release-note loop 2026 · GitHub
Pattern library
Drafts release notes from merged commits, issues, and PRs with linked evidence. Project artifact
🔁 Model-routing loop 2026 · GitHub
Pattern library
Routes tasks across models on measured quality, latency, privacy, and cost. Project artifact

Core Loop Primitives

Feature-level building blocks you assemble a loop from: schedulers, goals, worktrees, hooks, skills, plugins, and protocols.

Resource Published at Contribution Evidence
📚 Scheduled tasks - ChatGPT Learn ChatGPT Learn Official guidance for recurring background tasks, triage inboxes, skills, and isolated workspaces. Official documentation
📚 Follow a goal - ChatGPT Learn ChatGPT Learn Official guidance for durable objectives with stopping conditions, validation commands, checkpoints, and progress logs. Official documentation
📚 Git worktrees - ChatGPT Learn ChatGPT Learn Official worktree model for isolated parallel tasks and handoffs between local and background workspaces. Official documentation
📚 Prompting - ChatGPT Learn ChatGPT Learn Explains the Codex loop, threads, context, and goal-oriented prompting. Official documentation
📚 Customization overview - ChatGPT Learn ChatGPT Learn Maps AGENTS.md, memories, skills, MCP, and subagents into a coherent customization stack. Official documentation
📚 Build skills - ChatGPT Learn ChatGPT Learn Official skill format for reusable workflows, scripts, MCP dependencies, invocation policy, and plugin packaging. Official documentation
📚 Plugins - ChatGPT Learn ChatGPT Learn Bundles skills, app integrations, and MCP servers into reusable loop capabilities. Official documentation
🧰 dotskills 2026 · GitHub
vincentkoc/dotskills
A .skills registry of curated Codex and OpenClaw skills, framed as an "ADE Loop" (Agent Development Environment to registry to Skills Gym) where reusable skills are developed, shared, and evaluated across runs. Source implementation
📚 Developer commands - ChatGPT Learn ChatGPT Learn CLI commands for switching agent threads, browsing skills, inspecting MCP tools, and using subagent workflows. Official documentation
🔁 Autonomous Loops claudecodeguide.dev Claude Code pattern using task files, stop hooks, restart behavior, hard limits, and a kill switch. Operational pattern
📚 Claude Code Glossary Anthropic Defines the agentic loop, hooks, subagents, skills, MCP, and related primitives in Claude Code terminology. Official documentation
📚 Keep Claude working toward a goal Claude Code Docs /goal runs turn after turn until a completion condition is met by a verifier. Official documentation
📚 Run prompts on a schedule Claude Code Docs /loop, scheduled tasks, reminders, monitor tools, and session-scoped recurring prompts. Official documentation
📚 Automate work with routines Claude Code Docs Claude Code routines: persistent cloud automations triggered by schedules, API calls, or GitHub events, with connectors, scoped environments, and branch-push limits. Official documentation
📚 Desktop scheduled tasks Claude Code Docs Local recurring runs on your own machine, with the persistence, file-access, permission, worktree, and missed-run trade-offs that distinguish them from /loop and cloud routines. Official documentation
📚 Run parallel sessions with worktrees Claude Code Docs Worktree isolation for parallel sessions and subagents so concurrent edits do not collide. Official documentation
📚 Automate actions with hooks Claude Code Docs Claude Code hooks guide for deterministic lifecycle control around model actions. Official documentation
📚 Hooks reference Anthropic Event-level reference for session, turn, tool-call, and subagent hooks. Official documentation
📚 Common workflows - Claude Code Claude Code Docs Practical workflows for worktrees, subagents, CI, batch processing, planning, and resuming prior work. Official documentation
📚 Manage multiple agents with agent view Anthropic Dashboard for dispatching, monitoring, and attaching to background agent sessions. Official documentation
📚 Run agents in parallel Anthropic Compares agent view, subagents, agent teams, worktrees, tasks, and workflows for parallel work. Official documentation
📚 Orchestrate subagents at scale with dynamic workflows Claude Code Docs Moves loop state and branching into workflow scripts so large tasks do not overload the conversation context. Official documentation
📚 Create plugins Claude Code Docs Packaging model-invoked skills, agents, hooks, MCP servers, monitors, and settings as shareable loop components. Official documentation
📚 Model Context Protocol Model Context Protocol Standard protocol for exposing tools and data sources to agent loops. Official documentation
📚 Allowing GitHub Copilot CLI to work autonomously GitHub Docs Copilot CLI autopilot mode plus /every and /after scheduling, turning the CLI into an unattended loop that runs steps until a task is complete. Official documentation
🧰 opencode-scheduler 2026 · GitHub
different-ai/opencode-scheduler
OpenCode plugin that runs recurring agent jobs through OS-native schedulers (launchd on macOS, systemd on Linux), with workdir-scoped jobs, timeouts, and skipped ticks when the previous run is still active. Source implementation
🧰 Agent-Loop-Skills 2026 · GitHub
gaasher/Agent-Loop-Skills
Reusable verification-gated loops (autoresearch, scientific writing, data analysis, code and prompt optimization, red-teaming) packaged as open-standard Agent Skills, each with a feedback signal, run ledger, and termination conditions. Source implementation
🧰 launch-your-agent 2026 · GitHub
anthropics/launch-your-agent
Anthropic's official Claude Code skill set that operationalizes an interview, launch, grade-against-definition-of-done, iterate, and schedule loop for Claude Managed Agents, leaving a live recurring scheduled agent plus an eval scaffold and roadmap. Source implementation
🧰 kanban-md 2026 · GitHub
antopolskiy/kanban-md
File-based kanban board for autonomous agent loops: a CLI and TUI over plain Markdown files that gives multi-agent work a durable, inspectable work-intake queue. Source implementation
📝 Why Checkpoints Aren't Durable Execution 2026 · Diagrid
Yaron Schneider
Draws the line between saving state and guaranteeing completion, then walks LangGraph, CrewAI, and Google ADK through concrete gaps: no automatic failure detection, manual resume requiring the right thread or invocation id, duplicate execution when two processes resume at once. Practitioner analysis
📄 Agent Contracts: A Formal Framework for Resource-Bounded Autonomous AI Systems 2026 · arXiv
Qing Ye & Jing Tan
Extends the 1980 Contract Net metaphor from task allocation to resource-bounded execution, noting that modern agent protocols standardize connectivity but none bound how much an agent may consume or how long it may run. Contracts carry explicit lifecycle states and conservation laws that keep a delegated budget inside its parent's. Research preprint
📄 Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry 2026 · arXiv
Weiyang Guo et al.
Retry policy between two bad options: a full retry pays the whole interaction cost again, while retrieving past experience dilutes the signal. Identifies the pivotal erroneous turn and restarts only from that state, reusing the correct prefix and concentrating the learning signal near the error boundary. Research preprint
📝 Your Agent Needs a Computer, Not a Container 2026 · Cloudflare Blog Argues there is nowhere near enough compute for every user's agent to hold its own container, then splits the difference: one SQLite-backed virtual filesystem inside a Durable Object, reachable from both a cheap isolate and a full Linux container, so work moves between them over the same files. Targets containers for under 10% of agent work. Practitioner analysis
📄 Thinking Costs Tokens: When More Structure is Worth the Price 2026 · arXiv
Thomas Nolasque et al.
Asks whether a token budget exists below which planning and verification cost more than they return. Compares a single call against a verified search architecture across 14 budget tiers from 250 to 42,000 tokens over 28,000 completed cells, so the structure-versus-budget tradeoff is measured rather than assumed. Research preprint
📄 Rollback the World, Keep the Reflection: Rollback-Induced Reflection for Long-Horizon LLM Agents 2026 · arXiv
Yi Yu et al.
Treats recovery as rollback-boundary control rather than a choice between repairing context and restoring state: deciding when to intervene, where to resume, and what should survive the rollback. Existing methods either leave the altered environment in place or discard the experience that would prevent repeating the mistake. Research preprint
📄 Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents 2026 · arXiv
Jiapeng Li
When a write times out, the action may already have landed, so a blind retry duplicates it and giving up skips required work. Asks whether exactly-once belongs in the model, the harness, or the tool contract, using six services with twelve injected fault modes and 25,930 episodes graded against a ledger of committed effects. Research preprint
📄 TurnCost: Turns, Not Observations: Cache-Aware Loop Engineering for Coding Agents 2026 · POVC '26: 1st International Workshop on PromptOps and Vibe Coding 2026
Pramod Kumar & Raj Kumar Pandey
Across 600 coding-agent trajectories, turn count explains 84% of the variance in log cost against 4% for observation size, and 98.6% of input tokens hit the prefix cache. Derives when delegating to a cheaper model actually raises cost. Research paper

Official Runtime Guides

End-to-end operating guides and release notes from the runtime vendors themselves: how each platform expects you to run recurring agent work.

Runtime map progressing from a nearby session to a local desktop schedule, isolated CI and cloud execution, and a recoverable durable worker

Resource Published at Contribution Evidence
📚 Run long horizon tasks with Codex OpenAI Developers OpenAI's runbook for plan-edit-test-observe-repair-document-repeat work, including specs, plans, status logs, and validation gates. Official documentation
📚 Best practices - ChatGPT Learn ChatGPT Learn Official best practices for context, AGENTS.md, MCP, skills, subagents, and automations. Official documentation
📚 Agents SDK OpenAI Developers OpenAI guide for agent orchestration, tool execution, approvals, state, guardrails, and observability. Official documentation
📚 Agents - OpenAI Agents SDK openai.github.io SDK primitives for agents, tools, handoffs, guardrails, and runner-managed loops. Official documentation
📚 Running agents OpenAI Developers OpenAI guide to turns, state, approvals, sessions, and continuation in the SDK runtime loop. Official documentation
📚 Integrations and observability OpenAI Developers OpenAI guide to MCP wiring and traces as the basis for debugging and evaluation loops. Official documentation
📚 Sandbox Agents OpenAI Developers Splits the harness control plane from the sandbox execution plane for long-running file and command work. Official documentation
📚 Guardrails and human review OpenAI Developers Approval and validation boundaries for sensitive agent actions. Official documentation
📝 ChatGPT Work and the Codex Desktop App OpenAI OpenAI's July 9, 2026 launch of ChatGPT Work, a GPT-5.6-powered agent that stays with a project for hours by breaking a goal into smaller steps and completing them independently across apps and files, alongside Codex merging into the ChatGPT desktop app with in-sidebar PR review and cross-repository projects, extending the long-horizon coding-agent loop pattern to general knowledge work. Practitioner analysis ‡
Resource Published at Contribution Evidence
📚 Building agents with the Claude Agent SDK Anthropic Claude SDK overview for tool-using agents, subagents, state, permissions, and streaming. Official documentation
📚 How the agent loop works Claude Code Docs Official walkthrough of the inner agent loop that outer recurring loops build on. Official documentation
📚 Extend Claude with skills Claude Code Docs Claude Code skill system for reusable loop instructions and assets. Official documentation
📚 Create custom subagents Claude Code Docs Claude Code custom subagents with isolated context, model choice, and tool permissions. Official documentation
📚 Writing effective tools for AI agents 2025 · Anthropic Anthropic's guidance on evaluating and improving tool specs using agentic loops and realistic tasks. Official documentation
📚 Introducing advanced tool use on the Claude Developer Platform 2025 · Anthropic Tool search, programmatic tool calling, and tool-use examples for scaling large tool libraries without flooding context. Official documentation
📚 Effective harnesses for long-running agents 2025 · Anthropic Anthropic's guidance for agents that work across many context windows: durable progress artifacts, environment setup, and self-verification. Official documentation
📚 Claude Code best practices Claude Code Docs Widely cited workflow guidance that underlies many recurring Claude Code loops. Official documentation
📚 Claude Managed Agents: Scheduled Deployments and Vaults 2026 · Claude Scheduled deployments for Claude Managed Agents, where each cron firing starts a fresh session to complete the task, plus environment-variable vaults that let sandboxed agents authenticate tools while the real secret attaches only at the network boundary. Official documentation
📝 Getting Started with Loops 2026 · Claude Official Claude Code team guide by Delba de Oliveira and Michael Segner (June 30, 2026) that defines loops as agents repeating cycles of work until a stop condition is met, categorizes turn-based, goal-based (/goal), time-based (/loop, /schedule), and proactive loops by trigger and stop criteria, and recommends encoding verification as skills with quantitative success criteria alongside token-spend management. Practitioner analysis
📚 Claude Code What's New, Week 28 2026 · Claude Code Docs Weekly digest whose loop-integrity features include auto mode blocking tampering with session transcript files and confirmation prompts before destructive operations in unattended runs. Official documentation
Resource Published at Contribution Evidence
📚 GitHub Agentic Workflows GitHub Agentic Workflows Repository automation that runs coding agents in GitHub Actions on events or schedules with guardrails. Official documentation
📚 Continuous AI githubnext.com GitHub Next's umbrella framing for CI/CD-style AI automation across the software lifecycle, the category that agentic workflows demonstrate. Official documentation
📚 About GitHub Copilot coding agent GitHub Docs GitHub's autonomous coding agent: assign an issue, the agent works in an isolated Actions-powered workspace, and a reviewable pull request comes back. Official documentation
📚 GitHub Copilot for Jira Is Now Generally Available 2026 · The GitHub Blog General availability of Copilot for Jira: delegate a Jira issue to the Copilot coding agent, monitor session progress inside the issue, and send follow-up instructions that continue the same draft pull request instead of starting a new one. Official documentation
📚 Copilot Agent Session Streaming (Public Preview) 2026 · The GitHub Blog Public preview that streams Copilot agent session activity, including prompts, responses, and tool calls, from cloud agents, the CLI, and IDEs to SIEM-compatible endpoints and a REST API, giving enterprises an audit trail for delegated agent work. Official documentation
📚 Security Reviews in the GitHub Copilot App 2026 · The GitHub Blog Changelog adding on-demand security reviews inside the Copilot app, so an agent's proposed changes can be scanned for vulnerabilities before they are merged. Official documentation
Resource Published at Contribution Evidence
📚 Cursor cloud agents Cursor Documentation Remote agents that work asynchronously in isolated environments and hand results back for review. Official documentation
📚 Cursor 3.8: Improvements to Cursor Automations Cursor Cursor 3.8 changelog introducing an /automate skill that configures an automation's triggers, instructions, and tools from a plain-language description, plus Slack emoji-reaction and five new GitHub event triggers for dispatching cloud agents. Official documentation
📝 Expanding Our Long-Running Agents Research Preview Cursor
Cursor Team
Cursor's research preview of days-long autonomous agents gated by an upfront human-approved plan and cross-checked by multiple agents, reporting merge rates comparable to standard agents on runs as large as 52 hours and 151k lines. Practitioner analysis
Resource Published at Contribution Evidence
📚 Jules Jules Google's asynchronous coding agent that plans, executes tasks in isolated cloud VMs, and returns reviewable diffs. Official documentation
📚 Devin Docs Devin Docs Documentation for a long-running autonomous software engineer with sessions, playbooks, knowledge, and review boundaries. Official documentation
📝 Amp: Agents, Anywhere 2026 · ampcode.com
Amp
Amp launches remote agent creation on any machine with shell access plus a headless runner mode that lets multiple agents run concurrently without a terminal UI. Practitioner analysis
📝 Amp: Right on Schedule 2026 · ampcode.com
Amp
Amp adds scheduled agent runs, letting recurring work fire on a cadence with results reported back, moving the platform from on-demand sessions toward standing loops. Practitioner analysis
📚 Claude Code What's New, Week 29 2026 · Claude Code Docs Weekly digest of Claude Code changes relevant to recurring agent work, following the Week 28 loop-integrity updates. Official documentation
📚 Copilot Code Review: Customization and Configurability 2026 · The GitHub Blog Changelog adding customization and configurability to Copilot code review, sharpening the automated review gate teams put between agent-written changes and merge. Official documentation
📝 Amp: Event Driven Orbs 2026 · ampcode.com
Amp
Amp's July 23, 2026 launch letting orbs react to outside events: a plugin registers a durable webhook endpoint, each incoming event's signature is verified, and the configured handler runs in an orb thread seeded with trusted event metadata, turning GitHub CI failures, Linear issues, and Discord messages into event-triggered agent loops that post results back to external tools. Practitioner analysis
📚 Scan Your Codebase for Vulnerabilities Claude Code Docs Anthropic's Claude Security plugin runs a multi-agent deep scan of a repository or diff inside a Claude Code session: dynamic workflows orchestrate agents that map the architecture, build a threat model, and hunt vulnerabilities, with independent verifier agents gating every finding before it reaches the report. Official documentation
📝 Agent Automation Controls in GitHub Issues 2026 · The GitHub Blog Public preview that puts approval gates on autonomous issue triage: agents driven by Agentic Workflows or the Copilot cloud agent suggest label, field, issue-type, assignee, and close actions with a stated rationale, high-confidence actions apply automatically while lower-confidence ones queue for human accept/decline, and every change carries an audit trail. Practitioner analysis
📝 Copilot Cloud Agent for Linear Is Now Generally Available 2026 · The GitHub Blog General availability of Copilot cloud agent for Linear: assign a Linear issue to the agent, which analyzes it, opens a draft pull request from an ephemeral GitHub Actions environment, streams progress to the Linear activity timeline, and accepts mid-task steering via comments, with model selection, custom agent, and branch controls. Practitioner analysis
📝 The New Rules of Context Engineering for Claude 5 Generation Models 2026 · claude.dev Blog
Thariq Shihipar
Official Anthropic post (Jul 24, 2026) by Thariq Shihipar (MTS): Anthropic removed over 80% of Claude Code's system prompt for Opus 5/Fable 5 with no measurable eval loss, and lays out five shifts, rules-to-judgment, example-to-interface tool design, progressive disclosure, concise tool descriptions, and manual-to-automatic memory systems, plus the /doctor command for auto-optimizing harness configs. Practitioner analysis
📚 Claude Cookbook Claude Cookbook Anthropic's hosted cookbook of runnable recipes for building with Claude, including agent loops with tool use, memory management, context editing, and evaluation harnesses, kept current with each model generation as the reference implementation source. Official documentation
📚 The 2026-07-28 MCP Specification 2026 · Model Context Protocol Blog Official MCP release announcement (Jul 28, 2026) by lead maintainers David Soria Parra and Den Delimarsky, publishing the 2026-07-28 spec as final after a ten-week release-candidate validation window. Official documentation
📚 MCP Tasks Extension Model Context Protocol Normative documentation for io.modelcontextprotocol/tasks (SEP-2663), promoted out of experimental core into an official extension with the 2026-07-28 spec and redesigned around durable handles instead of session-bound objects. Official documentation
📚 Antigravity CLI 2026 · GitHub
google-antigravity/antigravity-cli
Google's Antigravity CLI release of Jul 28, 2026 turns the agent into a programmable component rather than a terminal session. Official documentation
📚 Enterprise Managed Settings Now Apply to the GitHub Copilot App 2026 · The GitHub Blog GitHub changelog (Jul 27, 2026) extending managed-settings.json governance, previously limited to interactive CLI and VS Code clients, to the Copilot cloud agent, the autonomous background worker that opens PRs from issue assignments. Official documentation
📚 GitHub Copilot in Visual Studio Code, July 2026 Releases 2026 · The GitHub Blog The Agents window (public preview) gets the observability and isolation primitives that parallel agent operation needs. Sessions can now be started in a Git worktree, and notably for Copilot, Claude, or Codex sessions, not just Copilot's own, giving each concurrent agent its own checkout instead of contending over one working tree. Official documentation
📚 Durable Workflows in Microsoft Agent Framework 2026 · .NET Blog Adds durability to an agent workflow through hosting configuration rather than a rewrite: each executor becomes a durable activity with automatic retry, progress is checkpointed after every step, and a resumed run reads completed executors from the checkpoint store instead of re-running them. Official documentation
📚 Context Engineering in Deep Agents Docs by LangChain Documents the thresholds a production harness actually needs. Tool payloads over 20,000 tokens are written to the filesystem and replaced by a reference, summarization and offloading fire at 85% of the context window, subagents return only results, and a composite backend routes /memories to durable cross-thread storage. Official documentation

Research Foundations

Loop Engineering is new as a practice name, but it builds on years of agent-loop, feedback, planning, and self-correction research.

Resource Published at Contribution Evidence
📄 ReAct: Synergizing Reasoning and Acting in Language Models 2023 · International Conference on Learning Representations (ICLR)
Shunyu Yao et al.
Foundational reason-act-observe loop for tool-using language agents. Research paper
📄 Reflexion: Language Agents with Verbal Reinforcement Learning 2023 · Advances in Neural Information Processing Systems 36 (NeurIPS)
Noah Shinn et al.
Converts environment feedback into written reflections stored in memory for future attempts. Research paper
📄 Self-Refine: Iterative Refinement with Self-Feedback 2023 · Advances in Neural Information Processing Systems 36 (NeurIPS)
Aman Madaan et al.
Generate-feedback-refine loop where a model improves outputs over repeated passes. Research paper
📄 CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing 2024 · International Conference on Learning Representations (ICLR)
Zhibin Gou et al.
Uses tools to ground critique and correction rather than relying only on introspection. Research paper
📄 Tree of Thoughts 2023 · Advances in Neural Information Processing Systems 36 (NeurIPS)
Shunyu Yao et al.
Search over multiple reasoning branches; relevant when loop design needs exploration before committing. Research paper
📄 Graph of Thoughts 2024 · Proceedings of the AAAI Conference on Artificial Intelligence 38 (AAAI)
Maciej Besta et al.
Generalizes thought structures beyond chains and trees, useful for complex loop planning and aggregation. Research paper
📄 Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models 2024 · Proceedings of the 41st International Conference on Machine Learning (ICML)
Andy Zhou et al.
Combines search, action, and environment feedback for language agents. Research paper
📄 Voyager: An Open-Ended Embodied Agent with Large Language Models 2024 · Transactions on Machine Learning Research (TMLR)
Guanzhi Wang et al.
Demonstrates lifelong skill acquisition through iterative exploration, feedback, and a skill library. Research paper
📄 Generative Agents: Interactive Simulacra of Human Behavior 2023 · Proceedings of the 36th ACM Symposium on User Interface Software and Technology (UIST)
Joon Sung Park et al.
Introduces reflection and memory mechanisms for long-running agent behavior. Research paper
📄 Measuring AI Ability to Complete Long Software Tasks 2025 · Advances in Neural Information Processing Systems 38 (NeurIPS)
Thomas Kwa et al.
METR's task-length time horizon metric; grounds why loop budgets, checkpoints, and escalation matter as autonomous work gets longer. Research paper
📄 Reflection-Driven Control for Trustworthy Code Agents 2026 · AAAI Workshop on Trust and Control in Agentic AI (TrustAgent)
Bin Wang et al.
Elevates reflection from an external pass to an internal control loop that monitors the agent's decision path during generation and constrains risky steps with low overhead. Research paper
📄 Hyperagents 2026 · arXiv
Jenny Zhang et al.
Self-referential agents that fold task-solving and self-modification into editable programs, extending the Darwin Godel Machine toward open-ended self-improvement, the loop where an agent rewrites its own improvement mechanism across runs. Research preprint
📄 PARC: An Autonomous Self-Reflective Coding Agent for Robust Execution of Long-Horizon Tasks 2025 · arXiv
Yuki Orimo et al.
Hierarchical plan-execute-assess loops that detect and correct strategic errors during multi-hour autonomous runs. Research preprint
📄 When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents 2026 · arXiv
Lu Yan et al.
Measures how agents drift from intent when specifications arrive incrementally across a long loop, and proposes a mitigation that recovers most of the loss. Research preprint
🧰 Reflexion code 2023 · GitHub
noahshinn/reflexion
Reference implementation and experiments for verbal reinforcement loops. Source implementation
📄 Stop Hand-Holding Your Coding Agent: Engineering the Loops that Replace Step-by-Step Prompting 2026 · arXiv
Sandeco Macedo
Position paper that formalizes the loop specification (trigger, goal, verification step, stopping rule, memory) as a reusable artifact handed to an agent harness, with a taxonomy, a five-level verification ladder, and a hand-coded analysis of fifty real-world loops. Research preprint
📄 From Question Answering to Task Completion: A Survey on Agent System and Harness Design 2026 · arXiv
Jianyuan Guo et al.
Survey that decomposes the agent execution harness into six runtime responsibilities (observation, context, control, action, state, verification) and argues task performance emerges from the interaction of model, runtime, task structure, and evaluation rather than the model alone. Research preprint
📄 MOSS: Self-Evolution through Source-Level Rewriting in Autonomous Agent Systems 2026 · arXiv
Qianshu Cai et al.
Self-evolution loop where the agent rewrites its own source code, with each change anchored to a production failure and accepted only after deterministic replay verification with rollback, lifting a four-task mean grader score from 0.25 to 0.61 without human intervention. Research preprint
📝 METR Time Horizon 1.1 2026 · METR Blog Update to METR's time-horizon methodology, expanding the task suite to 228 tasks (31 at 8+ hours), migrating to the open-source Inspect framework, and revising the post-2023 capability doubling time to roughly 131 days. Practitioner analysis
📄 MetaSkill-Evolve: Recursive Self-Improvement via Two-Timescale Meta-Skill Evolution 2026 · arXiv
Zefeng Wang et al.
Two-timescale recursive self-improvement where a fast loop rewrites task skills from execution traces while a slow loop evolves the meta-skill governing improvement itself, gaining up to 23.5 points on OfficeQA, SealQA, and ALFWorld. Research preprint
📄 SkillOpt-Lite: Better and Faster Agent Self-Evolution via One Line of Vibe 2026 · arXiv
Yifei Shen et al.
Formalizes agent skill self-evolution as zeroth-order optimization and distills it into a minimal pipeline of file-system trajectory exploration, consensus attribute mining, and independent validation gating, letting a smaller model surpass larger ones on LiveMath and SpreadsheetBench. Research preprint
📄 Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops 2026 · arXiv
Mingguang Chen et al.
Survey of 1,250 arXiv papers from 2024-2026 organized along two axes, what a self-improvement loop improves and its degree of loop closure, separating bounded evaluable self-refinement from open-ended recursive self-improvement. Research preprint
📄 From Atomic Actions to Standard Operating Procedures: Iterative Tool Optimization for Self-Evolving LLM Agents 2026 · arXiv
Haipeng Ding et al.
EvoSOP has agents distill recurring execution trajectories into reusable standard operating procedures and iteratively optimize the toolset through a construction, merging, evaluation, and pruning lifecycle. Research preprint
📄 TTHE: Test-Time Harness Evolution 2026 · arXiv
Jun Nie et al.
Adapts LLM agents at test time by evolving a population of candidate harnesses (the executable control program around the model) from execution traces, using a label-free agentic proposer and judge to sustain improvements on text-to-SQL and competitive programming while flagging execution-derived proxy reliability as the key open challenge. Research preprint
📄 DeepSearch-World: Self-Distillation for Deep Search Agents in a Verifiable Environment 2026 · arXiv
Xinyu Geng et al.
Introduces DeepSearch-Evolve, where a deep search agent improves by self-distilling its own trajectories inside a deterministic 420K-task verifiable environment, progress verification, grounded reflection, and failure recovery replace teacher trajectories and sparse RL reward, lifting a 9B model to 31.2% BrowseComp and 61.5% GAIA. Research preprint
📄 What Makes a Good Bug Report for an AI Agent? 2026 · arXiv
Lara Khatib et al.
Statistical analysis of 433 issues plus controlled multi-model experiments showing LLM repair agents succeed more when bug reports carry reproduction scripts, fix suggestions, and fault-localization cues, while longer natural-language reports correlate with lower success, directly informing how a loop's work-discovery step should specify tasks before dispatching agents. Research preprint
📄 AutoPersonas: A Multi-Timescale Loop Engine for Open-Ended Persona Evolution 2026 · arXiv
Mengchen Li
Names self-locking as a runtime failure mode of continuing agent loops, where accumulated state and history pull generation toward stale repetition (over 95% rolling action-repetition across an eight-model 40-day stress test), and proposes a multi-timescale loop that admits divergent material only through evidence-governed absorption, cutting macro-theme repetition from 61.8% to 36.3%. Research preprint
📄 Agentic Data Environments 2026 · IEEE Data Engineering Bulletin 50(1)
Elaine Ang et al.
Vision paper from the IEEE Data Engineering Bulletin reframing data systems as the active execution environment agents operate in, spanning files, APIs, applications, and system state, arguing the substrate under recurring agent loops should both amplify agent capability and enforce safety guarantees that bound the cost of failure. Research paper
📄 Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation 2026 · arXiv
Chenyang Yang et al.
Meta agent maps observed failure modes to harness adaptation strategies, letting small-model agents recover ~90% of frontier-LLM performance at ~4% of the cost, the harness itself becomes the optimization target. Research preprint
📄 Inside the Skill Market: From Software Engineering Activities to Reusable Agent Skills 2026 · arXiv
Jialun Cao et al.
First large-scale empirical study of public agent-skill repositories and marketplaces, characterizing which software-engineering activities get packaged as reusable skills, their coverage across the development lifecycle, how they evolve, and how they are evaluated - an activity-centric map of the skills layer that agent loops compose (Cao, Cheung, et al., HKUST). Research preprint
📝 Harness Engineering for Self-Improvement 2026 · lilianweng.github.io
Lilian Weng
Lilian Weng's deep-dive arguing the harness, the system surrounding a base model that orchestrates execution, matters as much as raw intelligence for recursive self-improvement, with a taxonomy of harness components and failure modes. Practitioner analysis
📄 Compile, Then Page: Executable SOP Programs and a Capability-Gated Runtime 2026 · arXiv
Chenglin Yu et al.
Compiles safety-critical standard operating procedures into executable pseudo-code run by a program-guided stack machine that pages the active frame while the LLM does semantic execution, finding runtime guidance is capability-gated (it helps strong models and harms weak ones) across a six-model, seven-domain SOPBench study. Research preprint
📄 Mako: A Self-Evolving Agentic Operating System for Autonomous Web Exploitation 2026 · arXiv
Praneeth Narisetty & Shiva Nagendra Babu Kore
Industrial self-evolving agentic OS that treats exploit capability as a mutable, versioned kernel: the agent observes its own failures, synthesizes new capabilities, proves them against a live target, and hot-loads them back, a self-improvement loop with in-loop verification. Research preprint
📄 How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study 2026 · arXiv
Yunbo Lyu et al.
Mixed-methods study of how practitioners actually design software-engineering agents, surfacing the recurring loop, harness, and verification decisions teams make and where their mental models diverge from benchmark assumptions. Research preprint
📄 Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries 2026 · Transactions on Machine Learning Research (TMLR)
Yubo Li
Survey and taxonomy of how agent skill libraries are created, evaluated, retired, and reused over time, organizing the fast-growing self-evolving-skills literature into a lifecycle framework. Research paper
Resource Published at Contribution Evidence
📄 SIA: Self Improving AI with Harness & Weight Updates 2026 · arXiv
Prannay Hebbar et al.
Co-evolves a task agent's harness and model weights through a meta-agent, target agent, and feedback agent that evaluate outcomes and carry improvements across generations. Research preprint
📄 Self-Improvements in Modern Agentic Systems: A Survey 2026 · arXiv
Zhe Ren et al.
Survey from the Zhuge/Schmidhuber group that frames a modern agent as a foundation model coupled to an operational scaffold (prompts, memory, tools, control logic) and organizes self-improvement research by which scaffold component gets updated and what signal drives the update. Research preprint
📄 Self-Evolving Agent Harnesses via Gated Semantic Quality-Diversity 2026 · arXiv
Xiaotian Luo et al.
Evolves the agent harness (prompts, injected knowledge, runtime config) with model weights frozen: an LLM diagnoses failures and proposes patches while deterministic code owns all sampling and significance testing, and accepted patches populate a pathology-keyed quality-diversity archive, with sealed-test gains of 9-15.5 points across seven domains. Research preprint
📄 XScientist: A Git-Like Research Protocol for Long-Running Autonomous Scientific Discovery 2026 · arXiv
Jixiang Luo
Treats long-running autonomous research as a continuous observable pipeline: exploration recorded as portable checkpointed artifacts with code, outputs, and evidence links, plus repair loops and human-in-the-loop quality gates, so failed branches stay inspectable. Single-author work, content is on-target for long-horizon loop infrastructure but adoption is unproven. Research preprint
📄 Knowledge-Centric Self-Improvement 2026 · arXiv
Xuefei Julie Wang et al.
Proposes a knowledge-centric alternative to agent-centric self-improvement: agents stay generic and disposable while the persistent, improving object is a curated shared knowledge base that agents write evidence-grounded insights into and then distill, reporting higher solve rates at lower cost with knowledge that transfers across tasks and model families. Research preprint
📄 OpenForgeRL: Train Harness-native Agents in Any Environment 2026 · arXiv
Xiao Yu et al.
Open-source framework for end-to-end SFT/RL training of agents that run inside real inference harnesses (Claude Code, OpenClaw) via a lightweight proxy that serves the harness's model calls while recording them as training data, with Kubernetes-orchestrated distributed rollouts. Research preprint
📄 AREX: Towards a Recursively Self-Improving Agent for Deep Research 2026 · arXiv
Shuqi Lu et al.
Family of recursively self-improving deep-research agents built on the discovery-verification asymmetry: an inner research loop gathers evidence while an outer loop audits the answer constraint-wise, flags unresolved claims, and launches targeted follow-up research, with autonomously learned compression of interaction history. Self-improvement plus verification-in-the-loop, both core list themes. Research preprint
📄 Workflow-Localized Mechanism Learning: Attribution-Guided Repair and Knowledge Reuse for Structured Agent Skills 2026 · arXiv
Zibin Lin et al.
Optimizes Agent Skills (reusable procedural knowledge for frozen-model agents) by jointly resolving where a workflow failed, which mechanism caused it, and which third-party Skill knowledge to reuse locally: node-mechanism attribution pinpoints the failed node and smallest valid edit target, bounded patches apply the repair, and provenance- and scope-aware selection imports external knowledge, reaching 90.3 hard accuracy on SpreadsheetBench with skills that transfer to WikiTableQuestions. Research preprint
📄 Sample-Efficient Learning from Agent Experience 2026 · arXiv
Chenhui Gou et al.
Uses experience distillation to internalize an agent's own interaction histories into model weights, so in-context experience gains persist after the experience leaves the context window, matching RL-style improvement with far fewer environment interactions. Research preprint
📄 PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning 2026 · arXiv
Yipeng Shi et al.
Policy-centric training paradigm that reframes skills as a dynamic training-time scaffold for long-horizon agent RL: rollout groups from the latest policy become evidence cards, task-specific evaluation adjusts the context for subsequent rollouts, and guidance is pruned as the policy strengthens before the scaffold is discarded at deployment, improving over strong baselines by up to 18.6% on ALFWorld and WebShop while using 32.1% fewer prompt tokens on search-augmented QA. Research preprint
📄 From Agent Failures to Text Policies: What Works and What Breaks 2026 · arXiv
Jaideep Ray & Ankit Goyal
Applies TextGrad-style natural-language policy learning to agent failure trajectories and separates two abilities usually conflated: following a useful text policy versus learning one from experience. Research preprint
📄 The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents 2026 · arXiv
Darshan Tank & Baran Nama
Nearly 6,000 runs across two office-automation benchmarks and three model harness stacks measuring what average success rate hides: skills also cause regressions (tasks solved without the skill, failed with it). Finding that should change how people ship Skills -- the best-performing skills win primarily by regressing less, not by gaining more. Research preprint
📄 Learning on the Job: Continual Learning from Deployment Feedback for Frozen-Weights Agents 2026 · arXiv
Valentin Tablan et al.
Shows ordinary production feedback -- one-bit outcome verdicts and after-the-fact corrections -- is a sufficient signal for continual learning when a frozen model is paired with external memory distilling each episode into retrievable natural-language rules. Research preprint
📄 Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills 2026 · arXiv
Siyuan Huang et al.
Names the central dilemma of self-evolving training loops: environment-bound methods get precise feedback but stay narrow, while open-ended self-generation broadens the task space and loses reliable verification, letting misleading rewards pollute the training loop. Research preprint
📄 Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning 2026 · arXiv
Shujin Wu et al.
Hypothesizes that AlphaEvolve-style test-time self-evolution succeeds because of meta-skills -- notably self-reflection against environment feedback -- that conventional post-training neglects entirely. Research preprint
📄 From Execution to Capability: Scientific Experience Consolidation via Procedural Knowledge Synthesis 2026 · arXiv
Liwei Dong et al.
Studies why executable feedback on one task rarely becomes durable capability on the next, and names the two obstacles precisely: trajectory-derived artifacts often encode source-specific repairs rather than cross-task mechanisms, and a weaker target model may not be able to operationalize a valid abstract procedure (the abstraction-execution gap). Research preprint
📄 Agent-UCT: Upper Confidence Bounds Applied to Trees for Agentic Workflow Optimization with Cost-Awareness 2026 · arXiv
Yang Li et al.
Scaffold optimization as tree search: extends UCT with a reuse-aware regularization term derived from a bipartite prefix reuse graph, biasing selection toward branches that reuse already-materialized configuration prefixes so the search stops re-executing shared pipeline stages under tight evaluation budgets. Research preprint
📄 The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation 2026 · arXiv
Tianyi Men et al.
Builds a unified controlled multi-turn environment to study where long-horizon planning ability actually comes from, which opaque internet-scale training makes impossible to isolate. Research preprint
📄 Context Assembly as the Controlled Variable: A Control-Theoretic View of Harness Policies for Frozen LLM Agents 2026 · arXiv
Debjyoti Paul
Formalizes the loop as a frozen inner model wrapped by an outer context policy, making prompt templates, demonstrations, and retrieved context the controlled variable, with stability guarantees and uncertainty calibration for online updates. A rare attempt to give harness engineering actual control-theoretic foundations. Research preprint
📄 Towards an Agent Operating System - Lessons from Classical and Cloud OS 2026 · arXiv
Gosia Steinder & Hubertus Franke
Argues agentic systems are pre-standardization and proposes deriving abstractions by extending classical OS and cloud primitives to stochastic, natural-language-mediated execution, the way POSIX and Kubernetes consolidated their eras. A useful frame for where loop infrastructure is heading. Research preprint
📝 Discovering Cryptographic Weaknesses with Claude 2026 · Anthropic Anthropic research report (2026-07-28) that doubles as one of the most detailed first-party accounts of a multi-day autonomous agent loop. Practitioner analysis
📄 Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering 2026 · arXiv
Junlin Yang et al.
Builds OpenMLE around four composable operators, Draft, Improve, Debug, Crossover, that compose into long-horizon search strategies, and trains a 35B execution-grounded model on them. Research preprint
📄 Rehearse: Stepping Back from the Confidence Cliff in Self-Improving Autoresearch 2026 · arXiv
Jiazhen Ji & Shouhong Ding
Names and measures the 'confidence cliff': as a self-improving autoresearch system accumulates successful modifications, its ability to predict which next idea will work collapses from ~83% to ~57%, so the loop degrades exactly when it looks healthiest. Research preprint
📄 Harness-G: A Graph-Structured Harness for Search Agents 2026 · arXiv
Yanning Hou et al.
Diagnoses 'retrieval-equivalence collapse' in RL-trained search agents: during Search-R1 training, rollouts for the same question keep emitting distinct query strings while their accumulated evidence sets increasingly overlap, so trajectories converge to utility equivalence and within-group returns carry almost no retrieval contrast to learn from. Research preprint
📄 Self-Harness: Harnesses That Improve Themselves 2026 · arXiv
Hangfan Zhang et al.
Turns harness design into a loop the agent runs on itself: mine model-specific failure patterns from execution traces, propose minimal harness edits tied to those failures, then accept an edit only when it survives regression testing. Argues good harness design is model-specific and so scales poorly by hand. Research preprint
📄 DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution With Demonstrations 2026 · arXiv
Lirong Che et al.
Treats harness evolution as sample-efficient adaptation: change the executable structure around a frozen model instead of its weights. Argues reward-only search breaks down in long-horizon stochastic settings where failures cannot be attributed to a mechanism, and bootstraps from human trajectories. Research preprint
📄 From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws 2026 · arXiv
Mengzhuo Chen et al.
Argues outcome-driven harness repair produces broad, poorly scoped edits because it never localizes the fault. Compiles execution traces and harness artifacts into an intermediate representation so a repair can be tied to the mechanism, such as tool interface or lifecycle, that caused the failure. Research preprint
📄 AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design 2026 · arXiv
Yaxin Luo et al.
Meta-harness optimizer that guides a code agent to recursively rewrite its own harness from rollout feedback, against the observation that existing harness paradigms stay static instead of accumulating reusable experience. Instantiated on paper-to-poster generation with the accompanying PosterBench. Research preprint
📄 Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model 2026 · arXiv
Stephanie Jarmak
Monograph on the gap between evaluating coding agents as models and deploying them as systems, synthesizing 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 system case records. Treats evaluation and operation as a dependency chain where a gain at one layer often fails to reach end-to-end outcomes. Research preprint
📄 Stop Comparing LLM Agents Without Disclosing the Harness 2026 · arXiv
Yunbei Zhang et al.
Formalizes the Binding Constraint Thesis: for long-horizon tasks across models of comparable frontier capability, performance variance is governed more by harness configuration than by model choice, so evaluation protocols systematically misattribute harness-level gains to model improvements. Treats the harness as a closed-loop controller. Research preprint
📄 HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution 2026 · arXiv
Wen Jiang et al.
Names three failure modes in self-evolving harnesses: credit assignment failure, where terminal success or failure leaves the responsible step ambiguous; shortcut learning, where the agent memorizes task-specific patterns instead of capability; and catastrophic forgetting from unguarded updates. Gates each accepted change. Research preprint
📄 How Do Agent Harnesses Create Value? Planning Information and Release Control in Stateful LLM Agents 2026 · arXiv
Yukun Zhang et al.
Isolates what harness guidance actually contributes by pairing task-specific plans against shuffled policy text matched word for word. Across 265 matched cells the real plans add 7.17 points of verified success, concentrated in harder tasks, and a read-only verifier rejects 61% of invalid episodes while withholding 17% of correct ones. Research preprint
📄 Grow the Harness, Not the Context: From Strategy-Free Scaffolds to Reusable Specialist Agents 2026 · arXiv
Laizhen Li et al.
Asks whether recurring control decisions can become reusable harness code instead of being rebuilt in every task's context. Starts from a scaffold with no task-solving controller, localizes each failure to a bounded code surface from execution traces, and gates repairs on held-out success with rollback. Research preprint

Model-Level Recurrence

Use these works to understand the inner computation of the model that an agent may call. Looped, recurrent-depth, and recursive models reuse learned computation within one inference to refine hidden states, vary effective depth, or simulate dynamics. Operational Loop Engineering sits outside that inference and decides when work runs, what tools and context are allowed, what evidence proves progress, what state survives, and when to retry, escalate, or stop.

Question Model-level recurrence Agent or operating loop
What repeats? A shared layer, block, module, or latent-state update A model/tool/action cycle or a complete agent run
Where is state? Hidden activations or learned latent state Context, files, checkpoints, traces, queues, and systems of record
What controls depth? Fixed recurrences, learned halting, routing, or compute budget Trigger policy, verification result, retry budget, escalation, and exit
What proves success? Task accuracy, loss, calibration, efficiency, or rollout fidelity External tests, evals, receipts, approvals, service thresholds, or human review
How do they connect? Supplies a model with stronger or more adaptive internal computation Governs how that model acts repeatedly in a real environment

Every entry below is tagged Model layer · Adjacent foundation in the README and exported as loop_layer=model, scope_fit=adjacent in the dataset.

Foundations And Theory

Resource Published at Contribution Evidence
📄 Universal Transformers
Model layer · Adjacent foundation
2019 · International Conference on Learning Representations (ICLR)
Mostafa Dehghani et al.
Introduces recurrent depth for Transformers by repeatedly applying shared self-attention and transition blocks, with optional per-position adaptive halting; establishes the architectural foundation for later looped models. Research paper
📄 Looped Transformers as Programmable Computers
Model layer · Adjacent foundation
2023 · International Conference on Machine Learning
Angeliki Giannou et al.
Constructs a constant-depth looped Transformer that advances an in-state program counter and executes reusable instructions, showing how iterative algorithms and in-context gradient descent can be represented through repeated shared computation. Research paper
📄 Looped Transformers are Better at Learning Learning Algorithms
Model layer · Adjacent foundation
2024 · International Conference on Learning Representations (ICLR)
Liu Yang et al.
Trains input-injected looped Transformers for in-context data fitting and shows that iterative shared computation can match standard Transformers on tested function classes with substantially fewer parameters. Research paper
📄 On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding
Model layer · Adjacent foundation
2025 · International Conference on Machine Learning
Kevin Xu & Issei Sato
Derives approximation rates for looped Transformers, identifies a loop-specific expressivity limit, and uses timestep-conditioned scaling to improve function approximation as recurrence increases. Research paper
📄 Reasoning with Latent Thoughts: On the Power of Looped Transformers
Model layer · Adjacent foundation
2025 · International Conference on Learning Representations (ICLR)
Nikunj Saunshi et al.
Connects effective recurrent depth to reasoning, proves that looped models can simulate multi-step chain-of-thought in latent space under the paper's construction, and studies the trade-off between reasoning and memorization. Research paper

Latent Reasoning And Adaptive Compute

Resource Published at Contribution Evidence
📄 Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
Model layer · Adjacent foundation
2025 · Advances in Neural Information Processing Systems 38 (NeurIPS 2025)
Jonas Geiping et al.
Presents Huginn, a 3.5B recurrent-depth language model trained on 800B tokens whose shared core can be unrolled further at inference, with gains concentrated on reasoning tasks and support for adaptive compute and KV-cache sharing. Research paper
📄 Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation
Model layer · Adjacent foundation
2025 · Advances in Neural Information Processing Systems 38 (NeurIPS 2025)
Sangmin Bae et al.
Combines shared recursive layers with token-level routers so difficult tokens receive more depth while attention and KV caching are restricted to active tokens. Research paper
📄 Scaling Latent Reasoning via Looped Language Models
Model layer · Adjacent foundation
2025 · arXiv
Rui-Jie Zhu et al.
Introduces the Ouro family of pretrained LoopLMs, combining latent iteration, learned depth allocation, and large-scale pretraining to study recurrent depth as a scaling axis distinct from parameter count and generated reasoning tokens. Research preprint
📄 LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
Model layer · Adjacent foundation
2026 · International Conference on Learning Representations (ICLR)
Ahmadreza Jeddi et al.
Trains variable-length latent trajectories with time and step-size conditioning plus shortcut consistency, allowing one model to trade compute for quality across inference budgets without retraining. Research paper
📄 MoDr: Mixture-of-Depth-Recurrent Transformers for Test-Time Reasoning
Model layer · Adjacent foundation
2026 · International Conference on Learning Representations (ICLR)
Xiaojing Zhang et al.
Replaces a single recurrent reasoning path with dynamically routed LoRA branches, adding solution-space exploration and load-balanced routing to the Huginn-style depth-recurrent backbone. Research paper
📄 ChainGPT: Dual-Reasoning Model with Recurrent Depth and Multi-Rank State Updates
Model layer · Adjacent foundation
2026 · International Conference on Learning Representations (ICLR)
Yunao Zheng et al.
Combines within-layer multi-substep state updates, state-guided sparse attention, across-layer recurrence, and adaptive stopping to increase latent reasoning depth without extending visible chain-of-thought. Research paper
📄 Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models
Model layer · Adjacent foundation
2026 · International Conference on Machine Learning (ICML)
Tianyu Fu et al.
Learns when a token needs extra latent refinement, using a neural decider, depth-aware LoRA, and cross-iteration attention to avoid always paying for or being degraded by additional loops. Research paper
📄 Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
Model layer · Adjacent foundation
2026 · Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306
Sajad Movahedi et al.
Stabilizes very deep recurrence with pre-normalization and residual scaling, then uses latent-state convergence as the halting signal so the model allocates more iterations to harder Sudoku, maze, state-tracking, and ARC-AGI instances without a separate stopping head. Research paper

Scaling, Stability, And Mechanistic Evidence

Resource Published at Contribution Evidence
📄 Loop the Loopies!
Model layer · Adjacent foundation
2026 · arXiv
Zitian Gao et al.
Introduces the Loopie family of sparse looped Transformers (20B parameters with 2B active and 6B with 0.6B active); the authors' matched-compute experiments over 3.5T pretraining tokens overtake a reproduced vanilla 30B-A3B baseline after roughly 600B tokens, while post-training plus test-time scaling reaches 35/42 on IMO 2025 and 20.3 on IPhO 2025. Research preprint
📄 LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling
Model layer · Adjacent foundation
2026 · EMNLP 2026
Wenkai Chen et al.
Combines sparse expert routing with weight-shared recurrence through iteration-conditioned adaptive normalization and capacity balancing; under matched parameters, per-token FLOPs, and active-sublayer ratios, the reported 3B model beats its vanilla MoE control on 8 of 9 benchmarks by more than one point on average, with the advantage persisting at 9B. Research preprint
📄 Sparse Layers are Critical to Scaling Looped Language Models
Model layer · Adjacent foundation
2026 · arXiv
Ryan Lee et al.
Finds that dense looped models lose their scaling advantage while looped MoE models recover expressivity because routing diverges across repeated passes; also shows that loop boundaries form stronger early-exit points than arbitrary layers in standard Transformers. Research preprint
📄 A Mechanistic Analysis of Looped Reasoning Language Models
Model layer · Adjacent foundation
2026 · arXiv
Hugh Blayney et al.
Traces cyclic recurrence inside looped language models and finds that many studied layers converge to distinct fixed points, with attention behavior stabilizing as recurrent blocks repeat feedforward-like inference stages; tests how block size, input injection, and normalization shape those dynamics. Research preprint
📄 Parcae: Scaling Laws For Stable Looped Language Models
Model layer · Adjacent foundation
2026 · Learning to Iterate Workshop at ICLR 2026
Hayden Prairie et al.
Treats the recurrent block as a dynamical system, constrains its stability, and studies how training and test-time recurrence trade parameters for additional computation. Research paper
📄 SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion
Model layer · Adjacent foundation
2026 · arXiv
Chengting Yu et al.
Recurs over progressively compressed representations so repeated layers specialize across resolutions instead of recomputing every token at full resolution. Research preprint
📄 Training-Free Looped Transformers
Model layer · Adjacent foundation
2026 · arXiv
Lizhang Chen et al.
Retrofits recurrence onto frozen pretrained models by applying a damped mid-stack block as smaller refinement steps, testing when inference-time looping helps without fine-tuning. Research preprint
📄 Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
Model layer · Adjacent foundation
2026 · arXiv
Harsh Kohli et al.
Shows systematic generalization and depth extrapolation in controlled recurrent-depth reasoning tasks while documenting overthinking when recurrence exceeds the useful computation horizon. Research preprint
📄 DeepLoop: Depth Scaling for Looped Transformers
Model layer · Adjacent foundation
2026 · arXiv
Shuzhen Li et al.
Derives residual-scaling rules that account for repeated parameter visits and tests them on GPT-style looped language models, addressing instability that nominal layer depth alone misses. Research preprint
📄 How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
Model layer · Adjacent foundation
2026 · arXiv
Kristian Schwethelm et al.
Separates recurrent passes from unique parameter depth and training compute to estimate when an additional loop is worth more than adding distinct layers. Research preprint

Code, World Models, And Architecture Bridges

Resource Published at Contribution Evidence
📄 LoopCoder: Scaling Code Intelligence via Looped Language Models
Model layer · Adjacent foundation
2026 · Findings of the Association for Computational Linguistics: ACL 2026
Jian Yang et al.
Scales looped code models through dense-to-loop initialization, recurrent pretraining, and post-training; the Findings of ACL 2026 paper reports a 40B-active/80B-total model trained on more than 12T code and general tokens. Research paper
📄 Looped World Models
Model layer · Adjacent foundation
2026 · arXiv
Hongyuan Adam Lu et al.
Introduces LoopWM, which repeatedly refines action-conditioned latent environment states with a parameter-shared Transformer core, stability constraints, and adaptive early exit for long-horizon simulation. Research preprint
📄 Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
Model layer · Adjacent foundation
2026 · arXiv
Ying Fan et al.
Introduces LOTUS, which loops over parallel latent thought blocks with intermediate supervision and reports explicit-chain-of-thought-level quality at 3B scale with lower thought-phase latency. Research preprint
📄 Looped SSMs: Depth-Recurrence and Input Reshaping for Time Series Classification
Model layer · Adjacent foundation
2026 · arXiv
Mónika Farsang et al.
Extends tied-depth recurrence beyond Transformers by repeatedly applying a shared state-space block and reshaping inputs across iterations, testing which loop principles transfer across model families. Research preprint
📄 Looped Diffusion Language Models
Model layer · Adjacent foundation
2026 · arXiv
Sanghyun Lee et al.
Applies selective shared-layer recurrence to masked diffusion language models so effective depth can scale during training and inference without adding parameters. Research preprint
📄 Metis: Memory Foundation Model
Model layer · Adjacent foundation
2026 · arXiv
Zeyu Zhang et al.
Argues for native memory inside the backbone rather than bolted-on external modules: a persistent memory state compresses history, is read through memory attention, and is maintained gradient-free with forward passes only at inference. The model-level answer to problems the harness-level memory stacks keep patching. Research preprint
📄 Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
Model layer · Adjacent foundation
2026 · arXiv
Rubin Wei et al.
Attacks the memory problem one layer below retrieval: decoder-only models entangle long-term memory and reasoning in a single parameter set, so memory capacity cannot be scaled independently of the reasoning capacity you are paying for. Research preprint

For a deeper architecture-only index, use Awesome Loop Models in the adjacent-list catalog below. Its stricter model definition complements this section; the resources above are the bridge most relevant to agents, coding systems, adaptive compute, and world simulation.

Agent Workflow Patterns

Use these resources to design the higher-level loop around agents: recurring control flow, delegation, verification, state, and next-action decisions.

Resource Published at Contribution Evidence
🧰 Discovery Loop Discovery Loop Company thesis for automating the experimental loop itself: propose, implement, run, examine, iterate, driven by frontier models over large-scale compute so thousands of experiments run in parallel instead of sequentially by hand. Starts with machine learning research and uses its own stack as the first customer. A statement of direction rather than a shipped system. Implementation
📚 Building Effective Agents 2024 · Anthropic Anthropic's canonical guide to workflows and agents, including evaluator-optimizer and orchestrator-workers patterns. Technical documentation
📄 Harness Engineering for Language Agents: The Harness Layer as Control, Agency, and Runtime 2026 · Preprints.org
Chaoyue He et al.
Decomposes the harness layer that loops build on into control, agency, and runtime, audits 63 harness works, and proposes a HarnessCard so reported agent gains can be separated from harness effects. Research paper ‡
📝 How we built our multi-agent research system 2025 · Anthropic Detailed orchestrator-worker system with planning, memory, subagents, citation passes, and iterative research loops. Practitioner analysis
🔁 12 Factor Agents 2025 · GitHub
humanlayer/12-factor-agents
Operating principles for production agents, including explicit prompts, state ownership, and pause-resume behavior. Operational pattern
🔁 Durable Execution for Agentic Workflows 2026 · Arizen Explains checkpointing, event-sourced journals, replay, and recovery for long-running agent workflows. Operational pattern
📄 Loop Engineering Meets Graph Engineering 2026 · GitHub
Lingjiao Chen et al.
Proposes Loop-Graph, a long-horizon agent framework that couples iterative refinement loops with graph-structured persistent memory; the supplied manuscript screenshot reports evaluation on 9,842 real-world tasks, success gains up to 38.6 percentage points, correctness gains up to 27.4 percentage points, fewer redundant tool calls, and lower latency versus strong prompting and RAG baselines. Research preprint
📄 Code as Agent Harness 2026 · arXiv
Xuying Ning et al.
Organizes agent infrastructure into harness interface, feedback-driven control, and multi-agent scaling for executable, verifiable, stateful systems; maps the harness layer that loops build on. Research preprint
📄 Agentic Agile-V: From Vibe Coding to Verified Engineering 2026 · arXiv
Christopher Koch
Proposes a task-level SCOPE-V loop (Specify, Constrain, Orchestrate, Prove, Evolve, Verify) with human approval gates, arguing agentic coding needs process control and independent verification, not better prompts. Research preprint
📄 Agentic Software Engineering: Foundational Pillars and a Research Roadmap 2025 · arXiv
Ahmed E. Hassan et al.
Splits agentic SE into an Agent Command Environment for human orchestration and an Agent Execution Environment for agent task execution, a research roadmap for the layers recurring loops run inside. Research preprint
📝 The Art of Loop Engineering LangChain LangChain's account of four stacked loops around agents (core execution, rubric-based verification, event-driven triggers, and trace-driven self-improvement) using a documentation-writing agent as the running example. Practitioner analysis
🧰 Loopy 2026 · GitHub
Forward-Future/loopy
Library of reusable AI-agent loops with verification checks and stopping conditions, plus an installable skill for finding, adapting, and designing repeatable agent workflows. Source implementation
📝 The Factory Model: How Coding Agents Changed Software Engineering 2026 · AddyOsmani.com
Addy Osmani
Addy Osmani's February 2026 essay recasting engineers as operators of parallel agent workstreams, with agents running "thirty minutes, an hour, several hours and increasingly days" and verification, not generation, as the new bottleneck; the fleet-level framing that precedes his June Loop Engineering essay. Practitioner analysis
📚 2026 Agentic Coding Trends Report 2026 · Anthropic Anthropic's official 2026 trends report on the shift from single coding assistants to coordinated agent teams running autonomously for hours or days, with case studies from Rakuten, TELUS, and Zapier (landing page is registration-gated; the direct PDF is public). Technical documentation
🧰 HomeRail 2026 · GitHub
xiaotianfotos/homerail
TypeScript runtime that turns one-off agent chats into auditable, reusable DAG workflows on self-hosted hardware, with a DAG engine, CLI, and voice front end. Source implementation
📝 Old and New Apps, via Modern Coding Agents 2026 · terrytao.wordpress.com Terence Tao's July 11, 2026 account of porting roughly two dozen Java 1.0 applets to JavaScript and resurrecting long-abandoned projects through iterative agent sessions, concluding that domain expertise remains the human verification layer: high-level design decisions stay with the author while implementation is automated away, and the exchange was "a net wash" on code quality, he caught one minor bug in the agent's output while the agent found two bugs in his original code. Practitioner analysis ‡
📄 Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable 2026 · arXiv
Ruhan Wang et al.
Introduces a behavior-centered source map and behavior-guided program decomposition for evolving harnesses, improving behavior localization and edit planning on two agent harnesses while reducing planner token use. Research preprint
📄 MemoHarness: Agent Harnesses That Learn from Experience 2026 · arXiv
Yue Huang et al.
Lets an agent adapt six harness dimensions per case using a dual-layer experience memory; results improve over fixed harnesses, while broad robustness and the contribution of each adaptive component remain open questions. Research preprint
📄 Recursive Harness Self-Improvement 2026 · arXiv
Hyunin Lee et al.
Represents an agent loop as a prompt-level harness specification and iteratively revises it from pairwise feedback over its own history; across 30 synthetic machine-learning research tasks, a few revisions lift low-reasoning agents above the corresponding maximum-reasoning setting while cutting inference cost by up to 60%, primarily through better context flow between agents. Research preprint
📄 SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents 2026 · arXiv
Yanze Wang et al.
Filters roughly 821,000 crawled agent skills into a 96,401-item corpus with a 16-class taxonomy and utility, robustness, and safety facets; evaluations across three benchmarks, two harnesses, and two open models report gains up to 7.5 percentage points while exposing both coverage and harness boundaries. The announced dataset, models, and code are pending release. Research preprint
📄 Behavioral Controllability of Agentic Models for Information Extraction: From Fixed Workflows to Reflective Agents 2026 · arXiv
Lujia Zhang et al.
Compares a fixed extraction workflow with reflective variants under one evaluation harness, measuring tool execution, retries, reflection, memory use, runtime, and recovery so agentic mechanisms can be judged by observable process changes rather than output scores alone. Research preprint
📄 Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports 2026 · arXiv
Jingyi Chen et al.
Manually analyzes 255 issue reports from Codex, Gemini CLI, LangChain, and CrewAI to classify bugs triggered by an interaction between model output and harness behavior; silent failures, weak test oracles, and stochastic reproduction motivate dedicated replay and fault-localization support. Research preprint
📄 SciForge: An AI-Native, Multimodal Workbench for Scientific Discovery 2026 · arXiv
SciForge Team et al.
Organizes long-running scientific work around goal-scoped review gates, domain translators, an agent runtime and workflow engine, and an evidence DAG that links claims to provenance; eight end-to-end cases include multi-day research sprints, while deeper team collaboration remains planned work. Research preprint
📝 Coding Agents 2.0: Interface, Inference, and Verification Gradient Ventures Argues the coding-agent bottleneck has shifted from generation to three infrastructure problems: interfaces for parallel agent fleets, a coding-optimized inference stack with prefix caching, and verification layers built on formal proofs and trace diffing. Practitioner analysis
📝 Towards a Harness That Can Do Anything eardatasci.github.io Essay on designing a general-purpose agent harness, examining what capability boundaries, tool surfaces, and verification hooks a do-anything harness actually requires. Practitioner analysis
📄 NVIDIA-labs OO Agents: Native Python Object-Oriented Agents 2026 · arXiv
Paul Furgale et al.
NVIDIA's model-agnostic framework (NOOA) where an agent is a Python object: methods are actions, fields are state, docstrings are prompts, and type annotations are contracts; a method body of '...' is completed at runtime by an LLM-driven agent loop while normal bodies stay deterministic, letting agent behavior be tested and refactored like ordinary software. Research preprint
🧰 deer-workflow 2026 · GitHub
deerwork-ai/deer-workflow
Code-first runtime that keeps orchestration and control flow in TypeScript while delegating only the semantic work to replaceable agent runtimes (Codex CLI and Claude supported out of the box). Emits an observable JSONL event stream for downstream automation and ships an interactive TUI for watching execution. Source implementation
📄 Living-Harness Is an Interactive-Agent Evolver 2026 · arXiv
Yuetian Du et al.
Converts each completed trajectory and its evaluator signals into posterior evidence for bounded harness updates, so the scaffold itself evolves via episodic memory and state graphs instead of staying static. Directly operationalizes the self-improving-loop premise: the harness is the learned artifact, not the weights. Research preprint
📄 CHILL-Harness: Counterfactual Harness Learning for Efficient Reasoning in Long-Horizon Agents 2026 · arXiv
Jiarun Fu et al.
Uses counterfactual causal learning to let an agent's harness adapt orchestration per task and environment, cutting long-horizon compute while holding success rate. One of the first papers to treat harness configuration as a learnable causal policy rather than a hand-tuned constant. Research preprint
📄 A Control System, a Dataset, and a Recipe for Making Frozen LLM Agents Learn a Domain 2026 · arXiv
Debjyoti Paul
RL-optimizes the harness around a frozen model by treating it as a fixed action space rather than allowing unconstrained self-modification, evaluated across tool-use workflows, code generation, and retrieval QA with released code and data. The bounded-action-space framing is the safety-relevant counterpoint to open-ended self-editing agents. Research preprint
📄 Scores Are Not Decisions: Cost-Aware Stopping for Tool Acquisition in LLM Agents 2026 · arXiv
Yicheng Feng et al.
Frames how many tools an agent should acquire as an optimal-stopping problem, training on the gap between stopping now and optimal continuation and proving the objective aligns with the stopping target under heterogeneous costs. Directly addresses when to halt, one of the least-formalized parts of loop design. Research preprint
🧰 Skill Recorder 2026 · GitHub
microsoft/skill-recorder
Microsoft's tool for authoring loops from demonstration instead of from prose. It captures a real work session locally, clicks, app and window switches, pages visited, optional spoken narration, then uses GitHub Copilot CLI to reconstruct what you actually did as one intent plus an ordered step list you review and edit. Source implementation
📄 SIGIL: Compiling Agent Skills into Typed Harnesses 2026 · arXiv
Jayanaka Dantanarayana et al.
Identifies that prose skill files force an agent to re-derive control flow on every run, then introduces 'skill compilation', lowering prose skills into executable harnesses through a typed IR (AG-IR). Reports 86% vs 56% of mandated steps executed, 2.3x more completed procedures, and 42% fewer tokens, stable across model generations. Research preprint
📄 RecHarness: A Bandit-Routed Agentic Harness for Self-Evolving Recommender Systems 2026 · arXiv
Haoran Ling et al.
Identifies a concrete instability in self-improving loops: when the LLM both picks the direction of a modification and generates the concrete hypothesis, search becomes unstable under limited experiment budgets. Research preprint
📄 DREvo: Distilling Recalibrated Historical Experience for Harness Self-Evolution 2026 · arXiv
Hanghui Guo et al.
Attacks the instability of harness self-evolution by anchoring evidence at function level, recalibrating which historical insights still apply given current state, and distilling role-conditioned search intent. Answers the practical question of when a past lesson should still steer the loop rather than mislead it. Research preprint
📄 SkillMentor: LLM Agent Self-Evolution via Learning Blind-Spot Diagnosis 2026 · arXiv
Xiaoyi Bao et al.
Treats blind-spot discovery as a learnable capability rather than assuming failures are already surfaced: an RL-trained Mentor policy generates diagnostic probe tasks, clusters recurring failure patterns, and converts them into reusable corrective skills without touching the executor or needing human labels. Research preprint
📄 A Decoupled Human-in-the-Loop System for Controlled Autonomy in Agentic Workflows 2026 · arXiv
Edward Cheng & Jeshua Cheng
Treats human oversight as an independent component of the agent operating environment rather than logic embedded in each workflow, on the observation that inlined approval code cannot be reused or kept consistent across a multi-agent estate. Separates interaction management from the application. Research preprint

Coding-Agent Loop Systems

End-to-end systems and field reports of coding agents running in loops, from single-repo Ralph runners to production agent fleets.

Resource Published at Contribution Evidence
🧰 SWE-agent 2024 · GitHub
SWE-agent/SWE-agent
Agent-computer interface and autonomous software engineering agent for repository tasks. Source implementation
📄 SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering 2024 · Advances in Neural Information Processing Systems 37 (NeurIPS)
John Yang et al.
Paper behind SWE-agent and its interface design. Research paper
🧰 mini-SWE-agent mini-swe-agent.com Minimal coding agent that is useful for understanding the core loop without a large framework. Implementation
🧰 OpenHands 2024 · GitHub
OpenHands/OpenHands
Open platform for AI software developers as generalist agents. Source implementation
📄 OpenHands: An Open Platform for AI Software Developers as Generalist Agents 2025 · International Conference on Learning Representations (ICLR)
Xingyao Wang et al.
Paper describing OpenHands, CodeActAgent, benchmarks, and generalist agent evaluation. Research paper
🧰 Agentless 2024 · GitHub
OpenAutoCoder/Agentless
Workflow-based approach for software issue resolution using localization, repair, and patch validation. Source implementation
📄 Agentless: Demystifying LLM-based Software Engineering Agents 2024 · arXiv
Chunqiu Steven Xia et al.
Useful contrast case: strong results through structured workflow rather than a fully open-ended agent. Research preprint
🧰 AutoCodeRover 2024 · GitHub
AutoCodeRoverSG/auto-code-rover
Autonomous program improvement system for issue localization, patch generation, and validation. Source implementation
📄 AutoCodeRover: Autonomous Program Improvement 2024 · Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)
Yuntong Zhang et al.
Paper on autonomous code repair loops over real repositories. Research paper
🧭 SWE-bench reading list 2025 · GitHub
SWE-bench/reading-list
Maintained map of software engineering agent systems and related papers. Discovery index
📄 TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Code 2026 · Proceedings of the 48th IEEE/ACM International Conference on Software Engineering (ICSE)
Jiangping Huang et al.
ICSE'26 observe-analyze-repair loop with instrumentation, analysis, and repair agents, a history-learning mechanism, and a rollback to the last good state; iteration alone drives most of the gain. Research paper
📄 The Kitchen Loop: User-Spec-Driven Development for a Self-Evolving Codebase 2026 · arXiv
Yannick Roy
Production loop where an agent exercises a spec surface as a synthetic power user behind ground-truth tests and quality gates, reporting 285+ self-correcting iterations and 1,000+ merged PRs with zero detected regressions. Research preprint
📄 Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures 2026 · arXiv
Benjamin Rombaut
Dissects 13 open-source coding-agent scaffolds and identifies five composable loop primitives (ReAct, generate-test-repair, plan-execute, retry, tree search) that real agents layer, mapping how control loop, tools, and state combine. Research preprint
📄 A Self-Improving Coding Agent 2025 · arXiv
Maxime Robeyns et al.
An agent that edits its own code and tools and re-runs against a benchmark, lifting itself from 17% to 53% on a SWE-bench Verified subset, a concrete self-modifying improvement loop. Research preprint
📄 Don't Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality 2026 · arXiv
Oussama Ben Sghaier et al.
Longitudinal study of 35 Qwen Code CLI releases with the model held constant, tracing coding-agent shifts to specific scaffolding changes in system prompts, tools, context management, and reasoning loops, and separating scaffolding regressions from model regressions. Research preprint
📄 ToFu: A White-Box, Token-Efficient Agent Harness for Researchers 2026 · arXiv
Junhao Ruan et al.
MIT-licensed white-box agent harness built on the thesis that agent behavior is set by the orchestration code around the model as much as the model itself, letting researchers inspect, modify, and evaluate its orchestration logic with reported token-efficiency gains over existing harnesses. Research preprint
📄 When Does Restricting a Coding Agent to execute_code Help? 2026 · KDD Workshop on Agentic Software Engineering (SE 3.0)
Hong Yang et al.
Regime-by-design ablation measuring when constraining a coding agent to a single execute_code action helps or hurts, separating task regime from agent design so harness choices can be made from evidence rather than intuition. Research paper
📄 Agentic Synthesis against Counterexample-Supplemented Sketches 2026 · arXiv
Muness Castle & Eric Rubeck
Turns human-approved failures into durable policy: a coding agent revises a code-shaped sketch, regenerates code and prompt surfaces, preserves provenance, and must pass a regression gate before seeing the next case. Research preprint
📄 Making Agent-Mediated Contributions Governable: A Project-Level Governance Manifest for Open-Source AI Collaboration 2026 · arXiv
Jinjin Gao et al.
Defines a repository-hosted governance contract connecting contributor evidence with maintainer verification and decision authority; after an audit of 50 repositories, a 15-reviewer study improves exact risk-label recovery from 15/37 to 37/38 with the manifest-supported materials. Research preprint
Resource Published at Contribution Evidence
🔁 Ralph 2025 · Geoffrey Huntley Geoffrey Huntley's original Ralph technique: run one agent in a bare loop with fresh context per iteration and the filesystem plus specs as memory. Operational pattern
🔁 everything is a ralph loop 2026 · Geoffrey Huntley Follow-up essay arguing the loop, not the agent, is the durable engineering unit: one task per iteration, deterministic context, and verification inside the loop. Operational pattern
🧰 The Ralph Playbook 2026 · GitHub
ClaytonFarr/ralph-playbook
Reference repository documenting the Ralph Wiggum technique end to end, from the bare loop script to guardrails and conventions. Source implementation
🔁 Ralph Copilot 2026 · GitHub
giocaizzi/ralph-copilot
Language-agnostic Ralph loop implementation using fresh context, filesystem memory, PRD.md, and PROGRESS.md. Operational pattern
🧰 Ralph (snarktank) 2026 · GitHub
snarktank/ralph
Ryan Carson's PRD-driven Ralph implementation that re-runs Amp or Claude Code with a fresh instance per iteration, gates each story on typecheck and tests, and persists state in prd.json, progress.txt, and Git history until every story passes. Source implementation
🧰 ralph-claude-code 2025 · GitHub
frankbria/ralph-claude-code
Loop runner that repeatedly re-executes Claude Code against project requirements, using dual-condition exit detection, rate limiting, and a circuit breaker to decide when the loop should stop. Source implementation
🧰 ralph-orchestrator 2025 · GitHub
mikeyobrien/ralph-orchestrator
Multi-backend implementation of the Ralph Wiggum technique that keeps a coding agent looping until task completion, using role-scoped hat personas that coordinate through events, with human-in-the-loop controls and a monitoring dashboard. Source implementation
🧰 ralphex 2026 · GitHub
umputun/ralphex
Extended Ralph loop runner that creates a Git branch per plan, executes tasks in fresh sessions with a commit after each, runs a multi-phase review pipeline with parallel review agents, and archives the completed plan. Source implementation
🧰 ralph (iannuttall) 2026 · GitHub
iannuttall/ralph
File-based Ralph-style agent loop that executes one JSON PRD story per iteration with fresh model context, using Git and on-disk state as memory across Claude, Codex, Droid, and OpenCode backends. Source implementation
🧰 ralph-loop-agent 2026 · GitHub
vercel-labs/ralph-loop-agent
Vercel Labs implementation of the Ralph loop for the AI SDK: an outer loop re-runs the agent with verifier feedback until a verifyCompletion check passes or iteration, token, or cost stop conditions trigger. Source implementation
🧰 Open Ralph Wiggum 2026 · GitHub
Th0rgal/open-ralph-wiggum
Agent-agnostic CLI that runs the Ralph Wiggum loop by feeding the same prompt to a fresh agent instance each iteration, with task tracking, live status monitoring, and mid-loop context injection across six coding-agent backends. Source implementation
Resource Published at Contribution Evidence
🔁 Compound Engineering every.to Every's named plan-work-review-compound loop, where each run feeds lessons back into AGENTS.md-style memory so the next loop is easier; the self-improving counterpart to Ralph. Operational pattern
🧰 Gas Town 2025 · GitHub
gastownhall/gastown
Steve Yegge's multi-agent orchestrator that runs 20-30 parallel coding agents with coordinator, worker, and merge-queue roles; the structured-orchestration end of the spectrum that Ralph anchors with bare iteration. Source implementation
🧰 Amp Amp Agentic coding tool built around threads, subagents, and an opinionated harness, with an owner's manual that documents loop-style operating practices. Implementation
🧰 karl 2026 · GitHub
kayoslab/karl
Autonomous multi-agent development loop with planner, reviewer, architect, tester, developer, deployment, and retry phases. Source implementation
🔁 joelclaw agent-loop skill 2026 · GitHub
joelhooks/joelclaw
Durable Planner-Implementor-Reviewer-Judge coding loops via Inngest events and progress files. Operational pattern
🧰 ARIS (Auto-Research-In-Sleep) 2026 · GitHub
wanshuiyin/Auto-claude-code-research-in-sleep
Markdown-only skills that run autonomous overnight ML research loops on Claude Code, Codex, or other LLM agents, iterating idea discovery and experiments with cross-model review as the verification gate. Source implementation
🧰 AutoAgent 2026 · GitHub
thirdlayerinc/autoagent
Meta-agent that autonomously edits its own harness (system prompt, tools, orchestration), re-runs the benchmark, and keeps or discards each change by score, with an author-reported top SpreadsheetBench result from a 24-hour unattended run. Source implementation
🧰 zeroshot 2025 · GitHub
the-open-engine/zeroshot
CLI that runs a planner, an implementer, and independent validators in isolated environments, looping until a change is verified or rejected with reproducible failures. Source implementation
🧰 Loki Mode 2025 · GitHub
asklokesh/loki-mode
Autonomous spec-to-app loop that runs Reason-Act-Reflect-Verify cycles behind quality gates, with completion gated by a blind three-reviewer council and a deterministic evidence receipt that rejects empty diffs and failing tests. Source implementation
🧰 Looper 2026 · GitHub
ksimback/looper
Claude Code skill for designing review-gated agent loops before running them, coaching the user into a portable loop.yaml spec with explicit goals, typed verification, iteration caps, and budget limits, then emitting artifacts runnable in-session or via an external Python runner. Source implementation
🧰 Agent Apprenticeship 2026 · GitHub
ray-r-ren/agent-apprenticeship
Multi-backend ecosystem where apprentice agents complete tasks through workflow loops, mentors or humans verify results, and execution traces are compiled into a published dataset that feeds future agent improvement. Source implementation
🧰 Scholar Loop 2026 · GitHub
renee-jia/scholar-loop
Autonomous multi-agent research loop from literature to hypothesis to real ML experiments to write-up, scoring every checkable agent claim against frozen ground-truth metrics and shipping an adversarial cheater engine that probes the loop for reward-hacking gaps. Source implementation
🧰 loop-engineering (Cobus Greyling) 2026 · GitHub
cobusgreyling/loop-engineering
Patterns-and-tooling repo shipping seven npm CLIs (loop-init, loop-audit, loop-cost, loop-sync, loop-context, loop-mcp-server, loop-worktree), starter kits, and production loop patterns; scaffolds skills/state/budget files, scores a repo's "Loop Ready" readiness, detects state drift, and estimates token spend per cadence for Claude Code, Codex, OpenCode, and Grok loops. Source implementation
🧰 AutoCVE 2026 · GitHub
larlarua/AutoCVE
Open-source agent-driven CVE discovery platform whose orchestrator coordinates Recon, Scan, Triage, Finding, and Verification agents through ReAct loops with correction nudges and structured FinalizeFinding termination, running the full discover, source-audit, dynamic-verify, dedup, and report loop on a self-hosted FastAPI/React/PostgreSQL stack with agent-tree observability. Source implementation
🧰 LoongFlow (Baidu) 2025 · GitHub
baidu-baige/LoongFlow
Baidu's open-source agent framework built explicitly for Loop Engineering: a Plan-Execute-Summary loop with structured experiential memory lets agents plan, execute, reflect, and evolve across software-engineering, math, and ML tasks (Apache-2.0, on PyPI, paper: arXiv 2512.24077). Source implementation
🧰 cc10x 2025 · GitHub
romiluz13/cc10x
Claude Code plugin that routes work through one router, nine specialist agents, sixteen skills, and four workflows, enforcing fail-closed verification and test-honesty gates and writing each workflow's intent, evidence, and verdicts to durable .cc10x/ disk artifacts so resume and review survive context compaction. Source implementation
🧰 RigorLoop 2026 · GitHub
ronikobrosly/RigorLoop
Statistically-grounded loop-engineering framework in which a strategy agent directs concurrent executor agents that iteratively build and refine a solution against gold-standard examples. Source implementation
🧰 Open-Inspect 2026 · GitHub
open-inspect/background-agents
Open-source background coding-agent system inspired by Ramp's Inspect: hosted agents in full dev-environment sandboxes, reachable from a web UI, Slack, GitHub PRs, Linear, or webhooks. Source implementation
🧰 T3MP3ST 2026 · GitHub
elder-plinius/T3MP3ST
Multi-agent offensive-security meta-harness that turns an existing coding agent into an autonomous vulnerability-research loop, with a verify-claims receipt step that separates confirmed findings from speculation. Source implementation
🧰 Loom 2026 · GitHub
valkor-ai/loom
Open-source delivery harness for existing coding agents that treats delivery as a durable loop: route, execute, verify, record evidence, repair, and continue from saved state. Source implementation
🧰 Inferoa 2026 · GitHub
agentic-in/inferoa
Inference-native agent harness for loop engineering that treats every loop as an inference workload, shaping each turn to preserve cacheable prefixes and bound stale evidence. Source implementation
🧰 PlanWeave 2026 · GitHub
GaosCode/PlanWeave
File-backed loop-engineering system for long-running coding agents that turns fuzzy plans into a claimable task graph of nodes and block documents routed through implementation and review. Source implementation
🧰 loop.js 2026 · GitHub
loop-js/loop.js
TypeScript loop-engineering framework that runs an agent in rounds against a stated goal until a skeptical, read-only verifier agent accepts the result or a budget is exhausted. Source implementation
🧰 ai-trains-ai 2026 · GitHub
Danau5tin/ai-trains-ai
Recursive training loop where a trainer agent autonomously writes complete reinforcement-learning jobs (environments, rewards, configs), runs them, and iterates on the results. Source implementation
Resource Published at Contribution Evidence
📝 Factory 2.0: From Coding Agents to Software Factories 2026 · Factory
Factory
Factory's software-factory pattern, where Automations coordinate recurring workflows with shared objectives and memory, Missions run multi-agent execution over hours or days, and Droid Computers give agents persistent remote execution across the SDLC. Practitioner analysis
📝 Superpowers 6 2026 · Massively Parallel Procrastination Release notes doubling as a case study of an unattended overnight autoresearch loop that ran 25 harness experiments against the project's own eval suite, roughly halving orchestration runtime and cutting token spend about 60%. Practitioner analysis
📝 Introducing Devin Security Swarm 2026 · cognition.com Cognition's agent swarm runs a continuous discover-verify-fix security loop: parallel agents hunt vulnerabilities, reproduce each in an isolated sandbox to confirm exploitability before reporting, and open remediation PRs, re-running on a schedule after the backlog clears. Practitioner analysis
📝 Towards Self-Driving Codebases Cursor
Wilson Lin
Cursor research on running thousands of coding agents as a recursive planner-subplanner-worker hierarchy sustaining roughly 1,000 commits per hour, finding that tolerating small error rates that peer agents later fix beats enforcing per-step correctness. Practitioner analysis
📝 Factory: Incident Response Automation 2026 · Factory
Factory
Factory's July 10, 2026 launch where a Droid triggered by Slack alerts (Sentry, Datadog, Rootly, Axiom) autonomously investigates each incident on a dedicated computer, triages, prepares fixes, and reports back in the thread, recording what it learns in a persistent runbook that Factory says improves its incident response over time. Practitioner analysis
📝 A Week-Long Autonomous Voxel Manhattan Build 2026 · X (formerly Twitter) Matt Shumer's demonstration of a single-prompt run in which a frontier model worked autonomously for almost a week with subagent fan-out to build a navigable voxel Manhattan. Practitioner analysis
📄 Self-Improving AI Coding Agents Through Accumulated Behavioral Rules: A Closed-Loop Framework 2026 · 32nd IEEE International Conference on Engineering Technology and Innovation (ICE/ITMC)
Aditya Aggarwal & Nahid Farhady Ghalaty
Converts accepted review feedback into versioned behavioral rules that future coding sessions can apply; across 11 reported production sessions, error classes covered by a rule did not recur, a promising but deliberately small-scope result. Research paper
Resource Published at Contribution Evidence
🧰 CUAWright 2026 · GitHub
microsoft/CUAWright
Treats workspace code, screenshots, and run artifacts as durable state while browsers remain disposable, producing a readable write-run-inspect-repair loop for long-horizon browser and desktop tasks. Formerly Webwright. Source implementation
📝 Porting Software Has Been Trivial for a While Now 2026 · Geoffrey Huntley Geoffrey Huntley, originator of the Ralph technique, on porting entire codebases by pointing a looped coding agent at the source and target and letting verified iterations do the work. Practitioner analysis
🧰 DeerFlow 2025 · GitHub
bytedance/deer-flow
ByteDance's open-source long-horizon SuperAgent harness that researches, codes, and creates across extended multi-step runs, one of the most widely adopted open agent harnesses. Source implementation
🧰 Grok Build 2026 · GitHub
xai-org/grok-build
xAI's official agentic coding CLI that plans, edits, and verifies changes in a repository, joining the major vendor coding-agent runtimes. Source implementation
🧰 BlitzOS 2026 · GitHub
blitzdotdev/blitzos
Open-source pattern for booting cloud agents pre-loaded with your work context so recurring sessions resume instantly instead of cold-starting. Source implementation
📝 Star Fleet Math: Solving Erdős Problems with 20 Parallel Codex Harnesses Star Fleet Math Field report on running twenty parallel Codex harnesses against open Erdős problems, a live demonstration of fleet-scale loop orchestration applied to mathematics research. Practitioner analysis
🧰 Finn-loop 2026 · GitHub
finna/Finn-loop
Three Claude Code skills from Alex Finn that turn Linear + GitHub into a small, human-gated software factory: /finn-spec interviews you until the behavior is unambiguous and files a Linear issue with acceptance criteria and non-goals, /finn-build claims the next agent-ready issue and opens a PR (run repeatedly via the /loop primitive), and /finn-review posts loop-approved or changes-requested verdicts against required CI checks, one approval label, one rule: humans merge. Source implementation
📄 How Do AI Coding Agents Contribute to Software Development? An Empirical Study of Agentic Pull Requests 2026 · arXiv
Iren Mazloomzadeh et al.
Mazloomzadeh, Morovati, and Khomh run a longitudinal analysis over the AIDev dataset covering 9,428 agentic PRs from five agents across 489 Python repositories. Research preprint
📄 Agent Team Work Zone: An Automated, Persistent Workspace for Long-Lived Coding Agent Teams 2026 · arXiv
Shouren Wang
Explicitly diagnoses four Claude Code Agent Teams failure modes and builds a persistent workspace to fix them: irrecoverable agent teams (state lost when the terminal closes), compaction eroding working detail, agentic 'technical debt' trapped in compacted old chats, and heavy handoff prompt writing. Research preprint
📄 Claim Plane: Enforceable Change Intents and Dynamic Scope for Parallel Coding Agents 2026 · arXiv
Maxim Nikolaev
Reframes parallel coding agents as a pre-write admission problem instead of a merge-time repair problem. Research preprint
📄 Plans Work in Mysterious Ways: Evaluating a Plan Mode for Spreadsheet Agents 2026 · arXiv
Aayush Kumar et al.
Plan Mode is now standard in agentic coding tools, but nobody has tested whether the transparency-and-control benefit survives outside developer contexts. Research preprint
🧰 JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents 2026 · arXiv Open harness for long-horizon multimodal production, built on the observation that real creative work is an evolving project state -- references, drafts, alternatives, edits, failed attempts, version relations, tool actions, evaluation signals, human feedback -- that prompt-based, chat-based, and node-based systems discard, linearize, or force users to wire manually. Research preprint
📄 MultiFixer: A Coordinator-Proposer Based Multi-Agent Framework For Fixing Multi-Hunk Bugs 2026 · arXiv
Haichuan Hu et al.
Coordinator-proposer architecture for bugs needing coordinated edits across multiple locations, combining tool-augmented analysis, fine-grained repair context construction, and iterative patch generation to reach state of the art on multi-hunk benchmarks. Targets the repair case single-agent loops most reliably fail. Research preprint
🧰 ECC 2026 · GitHub
affaan-m/ECC
Self-described agent harness operating system, packaging planning, execution, and optimization for coding agents behind one runtime with documentation in several languages. Very widely starred; treat adoption counts as a popularity signal rather than a quality judgement and read the source before relying on it. Source implementation
📝 Harness Design for Long-Running Application Development 2026 · Anthropic Engineering
Prithvi Rajasekaran
Anthropic Labs harness that splits planning, generation, and evaluation into separate agents so the critic is never the author. Reports a retro game maker going from a broken 20-minute solo run to a working build over six hours, and a browser DAW finished in 3h50m for $124.70. Practitioner analysis
🧰 LongHorizon-Harness 2026 · GitHub
AMAP-ML/LongHorizon-Harness
Runnable implementation of the Manage-Execute-Audit loop, driving agents across desktop apps and the CLI over long sessions with fresh-context execution, durable verified state, independent auditing, and recoverable progress. Integrates with Claude Code, Codex, and OpenClaw. MIT. Source implementation
🧰 DeepCode 2025 · GitHub
HKUDS/DeepCode
Multi-agent coding system built around verification-driven repair: a failed check becomes the input to the next attempt, and the run continues until evidence supports completion or the task is judged blocked. Sessions persist tool calls, permission decisions, and verification records for exact replay, and parallel agents work in isolated Git worktrees. MIT. Source implementation
🧰 OpenHarness 2026 · GitHub
HKUDS/OpenHarness
Deliberately small, readable harness for seeing how one works from the inside: 43 tools, on-demand skill loading, auto-compaction that carries task state across long sessions, and three permission modes from prompt-before-write through plan-only. PreToolUse and PostToolUse hooks expose the lifecycle. MIT. Source implementation
📝 Agent Harness Engineering 2026 · AddyOsmani.com
Addy Osmani
Defines the harness as everything that is not the model, then catalogues the mechanisms: compaction and tool offloading against context rot, planner/generator/evaluator splits because separate agents beat self-evaluation, hooks that block destructive commands, and a ratchet principle where every mistake becomes a rule. Practitioner analysis
🧰 YYLO 2026 · GitHub
yylo-dev/yylo
Command-line orchestrator that runs each coding-agent task in its own Git worktree bound to a recorded target, then gates completion on validation. Review depth scales with assessed risk, and the review cycle ends on an explicit REVIEW_FINDINGS_EXHAUSTED stop rather than looping indefinitely. MIT. Source implementation
📄 Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement 2026 · arXiv
Haoyang Yan et al.
Runs existing coding-agent harnesses inside iterative planning, coding, and testing loops for multi-day development without human intervention. Sustains improvement by balancing repair against capability growth and scoping work into small verifiable increments, with implementation-time tests kept separate. Research preprint

Verification And Feedback Gates

Turn harness and observability mechanisms into exit gates, durable receipts, and retry signals with the resources below.

Resource Published at Contribution Evidence
📝 Why Agentic Systems Must Produce Deterministic Outputs to Scale streamzero.com Argues for deterministic boundaries, contracts, and execution gates around probabilistic agent reasoning. Practitioner analysis
🔁 Stop Babysitting Your Coding Agent. Give It Backpressure. generativeprogrammer.com Explains how to turn tests, linters, builds, traces, and other signals into feedback loops for coding agents. Operational pattern
🔁 How to Build a Self-Verification Loop in Claude Code DEV Community Uses hooks to enforce syntax, intent, and regression checks before an agent can finish. Operational pattern
📝 Agentic Code Review 2026 · AddyOsmani.com
Addy Osmani
Addy Osmani argues that review, not code generation, is the bottleneck in agentic workflows, proposing risk-tiered verification depth, heterogeneous AI reviewers, and hard CI gates while warning against closed loops of models with correlated blind spots. Practitioner analysis
📝 How the Claude Code team does code review 2026 · claude.dev Blog
Addy Osmani et al.
One Claude Code PR ran 50-plus agent review rounds overnight; fixes such as a three-round cap and diff-only re-review cut PRs needing five or more rounds from 8.9% to 1.5%. Agent review loops need stop rules. Practitioner analysis
📝 Using DSPy to Evaluate and Improve Datasette Agent's SQL System Prompts 2026 · Simon Willison’s Weblog
Simon Willison
Simon Willison wires a DSPy evaluation harness to a live Datasette instance with real tool calls and gold-standard metrics, then uses the eval traces to find and fix weaknesses in the agent's SQL system prompt. Practitioner analysis
📝 Agentic coding notes danluu.com Dan Luu's first-hand benchmarks and workflows arguing that systematic test infrastructure such as fuzzing and randomized testing, not human review, is what lets agent-generated code ship, and documenting why a self-contained agentic quality loop has so far eluded him. Practitioner analysis
📝 Understanding Is the New Bottleneck 2026 · geoffreylitt.com Geoffrey Litt argues that human understanding, not verification, is the real bottleneck in agent loops, warning that cognitive debt accrues when iterations outpace comprehension and proposing literate diffs, quizzes, and interactive micro-worlds as speed regulators. Practitioner analysis
📝 Verifying Agentic Development at Scale 2026 · cognition.com
Ido Pesok
Cognition details the verification stack behind Devin sessions going majority-async: source-grounded test plans, deterministic reusable testing skills, and annotated video artifacts with pass/fail assertions so unattended runs return merge-ready results. Practitioner analysis
📝 Loop Engineering Without Verification Is Just Automation sonarsource.com Sonar formalizes a two-tier verification gate for agent loops, pairing a probabilistic LLM verifier sub-agent for intent with a deterministic analysis gate as the hard halt, arguing that LLM-only verification amounts to two optimists agreeing. Practitioner analysis
📝 Closing the Verification Loop: Observability-Driven Harnesses 2026 · Datadog
Alp Keles, Jai Menon, Sesh Nalla, Vyom Shah
Datadog engineers' case for harness-first engineering once agents write code faster than humans can review, using deterministic simulation testing across millions of seeds as the verification gate. Practitioner analysis
Resource Published at Contribution Evidence
📝 Better Harness: A Recipe for Harness Hill-Climbing with Evals LangChain LangChain's recipe for using evals as the learning signal for harness improvement. Practitioner analysis
📝 Improving Deep Agents with harness engineering LangChain Practical discussion of self-verification, traces, middleware, and loop detection for coding agents. Practitioner analysis
📄 Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses 2026 · arXiv
Jiahang Lin et al.
Closed loop that turns each harness edit into a falsifiable contract verified against trajectory outcomes, so the harness evolves from observability rather than trial and error. Research preprint
📄 Meta-Harness: End-to-End Optimization of Model Harnesses 2026 · arXiv
Yoonho Lee et al.
Optimizes the surrounding harness (tools, prompts, control flow) end to end against task outcomes, turning harness tuning into a measurable improvement loop instead of manual trial and error. Research preprint
🧰 HALO (Hierarchical Agent Loop Optimizer) 2026 · GitHub
context-labs/halo
Analyzes production agent traces to find harness-level failure modes, hands its report to a coding agent to apply fixes, and repeats the collect-analyze-fix-redeploy cycle, reporting AppWorld gains from harness changes alone. Source implementation
📄 Harness-Aware Self-Evolving: Co-Evolving Model Weights, Harness, and Task Solutions 2026 · arXiv
Haochen Luo et al.
Agentic RL framework in which one model both solves tasks and edits its own harness, including repairing faulty evaluation code, co-evolving weights, harness, and solutions so a trained Qwen3-8B matches a much larger baseline. Research preprint
🧰 auto-harness 2026 · GitHub
neosigmaai/auto-harness
Bring-your-own-agent framework for self-improving agentic systems that mines failures from runs, optimizes the harness in response, and gates every change behind regression checks. Source implementation
📝 How Kiro fixes itself, and how we keep the fixes honest Kiro Kiro's trace-mining loop opened 8 PRs in its first week and 7 merged. Acceptance is decided in code, apart from the proposer: a fix must pass 10 of 10 runs, against at most 4 of 10 without it. Practitioner analysis
Resource Published at Contribution Evidence
📚 OpenAI agent evals OpenAI Developers Evaluation guidance for moving from traces to repeatable grading of agent workflows. Official documentation
🧰 Promptfoo OpenAI Agents provider promptfoo.dev Testing and assertions for multi-turn agent workflows, tools, state, handoffs, sandboxes, and traces. Implementation
🧰 Inspect AI 2023 · GitHub
UKGovernmentBEIS/inspect_ai
UK AISI evaluation framework with solvers, scorers, sandboxing, tool use, MCP, and log viewing. Source implementation
📚 OpenTelemetry Semantic Conventions for Generative AI Systems OpenTelemetry Portable tracing conventions for model calls, tool calls, and agent workflows. Official documentation
🧰 AgentOps 2023 · GitHub
AgentOps-AI/agentops
Monitoring, replay, cost tracking, benchmarking, and tracing for agent sessions. Source implementation
🧰 Langfuse 2023 · GitHub
langfuse/langfuse
Open-source LLM engineering platform with tracing, evaluations, and metrics that loops can read back as feedback signals. Source implementation
🧰 LangSmith LangChain Tracing, evaluation, and monitoring platform for inspecting and grading agent runs across iterations. Implementation
🧰 Arize Phoenix 2022 · GitHub
Arize-ai/phoenix
Open-source AI observability for tracing, evaluating, and debugging agent behavior from real runs. Source implementation
🧰 Braintrust Braintrust Evaluation and observability platform with experiments, datasets, and CI integration for gating agent changes. Implementation
🧰 Weave CoreWeave Documentation Weights & Biases toolkit for tracing, evaluating, and monitoring agent applications over time. Implementation
🧰 agentops (boshu2) 2025 · GitHub
boshu2/agentops
Independent verification layer for coding agents where a change only counts as done after a different model or a real test checks it, with the verdict recorded in the repo via a tamper-evident ledger. Source implementation
🧰 SkillSpec 2026 · GitHub
modiqo/skillspec
CLI that makes agent skills followable, testable, and provable by converting prose skills into structured contracts, scoring follow-through risk, and generating execution traces of which steps ran, were skipped, and what evidence exists. Source implementation
🧰 Shepherd 2026 · GitHub
shepherd-agents/shepherd
Python runtime that records agent execution as reversible, Git-like traces so meta-agents or humans can observe, fork, replay, and revert any run before results touch files, with copy-on-write forking, roughly 95% cache reuse on replay, and syscall-level permission enforcement. Source implementation
🧰 grill-for-unknowns 2026 · GitHub
nicobailon/grill-for-unknowns
Portable SKILL.md agent skill that gates long-running subagent and coding-agent launches behind plan interrogation: it inspects the real territory (docs, source, tests, config) first, sorts what's known into facts, decisions, domain language, and unknowns across known/unknown quadrants, then emits a launch packet with assumptions, verification steps, and rollback risks before dispatch. Source implementation
🧰 Fable Harness 2026 · GitHub
Miguok/fable-harness
Drop-in behavior protocol kit (hooks, a skill, and sub-agents auto-injected into every Claude Code session) enforcing a verify-first process: gather evidence before answering and verify changes before declaring done. Source implementation
🧰 Mindwalk 2026 · GitHub
cosmtrek/mindwalk
Local visualization tool that replays Claude Code and Codex session logs as light moving across a 3D map of the repository, making long agent runs inspectable after the fact. Source implementation
🧰 Waggle 2026 · GitHub
modiqo/waggle
MCP-native reference layer for agent handoffs: instead of pasting full context between agents, it passes a compact attributed, resolvable reference token that the receiving agent expands on demand. Source implementation
🧰 Jacquard 2026 · GitHub
jbwinters/jacquard-lang
Research language designed around the machine-writes, human-verifies contract, using effect-typed signatures so an agent's generated code carries checkable declarations of what it is allowed to touch. Source implementation
Resource Published at Contribution Evidence
📄 Agentic Verification of Software Systems 2026 · Proceedings of the ACM on Software Engineering 3 (FSE)
Haoxin Tu et al.
Pairs a coding agent with a theorem prover (AutoRocq) in a generate-and-validate loop, turning formal proof into the exit gate for trusted automatic programming. Research paper
📄 A Trace-Based Assurance Framework for Agentic AI Orchestration: Contracts, Testing, and Governance 2026 · Proceedings of the 21st International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE)
Ciprian Paduraru et al.
Treats execution traces as the assurance substrate, pairing machine-checkable contracts, testing, and governance so recurring agent orchestration stays verifiable and auditable. Research paper
📄 Self-Evolving Agents with Anytime-Valid Certificates 2026 · arXiv
Biswa Sengupta
Confines self-modification to a small steering adapter around a frozen base model and gates each change with anytime-valid statistical tests that emit auditable certificates, reporting solve-count gains and logged regression prevention on a SWE-bench Verified subset. Research preprint
📄 Delayed Verification Destabilizes Multi-Agent LLM Belief 2026 · arXiv
Igor Itkin
Models verifier-corrector loops in multi-agent LLM systems as delayed consensus, deriving a stability threshold where verification that is too strong or too late turns factual consensus into oscillation, plus a greedy corrector-placement algorithm validated on five open models. Research preprint
📄 Lean4Agent: Formal Modeling and Verification for Agent Workflow and Trajectory 2026 · arXiv
Ruida Wang et al.
Models agent workflows and trajectories in Lean 4 dependent types so semantic consistency is machine-checked rather than judged by an LLM, with verification-passing workflows outperforming failing ones by an average of 11.94% on software-engineering benchmarks. Research preprint
📄 Regimes: An Auditable, Held-Out-Gated Improvement Loop 2026 · arXiv
Yohei Nakajima
Event-sourced agent runtime whose self-improvement loop gates every proposed repair behind static checks, sandbox execution, and held-out evaluation before adoption, keeping the full decision trail replayable. Research preprint
📄 Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents 2026 · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL)
Asaf Yehudai et al.
Automated evaluation framework from IBM Research that grades agent behavior at system, trace, and node granularity without predefined error taxonomies, producing feedback aligned with human-annotated errors and predictive of task success. Research paper
📄 Diagnosis-Driven Automatic Repair for Agentic Workflow via Symbolic Inference 2026 · arXiv
Xuyan Ma et al.
FlowFixer converts runs of platform-built agentic workflows (Dify, Coze, n8n) into symbolic traces, infers correctness specs and node dependencies to localize root-cause failures, and generates targeted repairs at a 71.3% success rate. Research preprint
📄 SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use 2026 · arXiv
Jiayin Zhu et al.
Self-evolving rubric framework that scores agent trajectories on skill selection, following, composition, and reflection, exposing failures that pass/fail outcome checks miss and beating outcome-only filtering as a training signal. Research preprint
📄 SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Bug Reproduction Tests 2026 · arXiv
Yaoqi Guo et al.
Shows that naively feeding bug-reproduction tests to software-engineering agents can mislead them, and instead pipes runtime diagnosis from multi-faceted reproduction tests into patch generation, reaching 75.7% on SWE-bench Verified. Research preprint
📄 AgentTether: Graph-Guided Diagnosis and Runtime Intervention for Reliable LLM Agent Operation 2026 · arXiv
Chenyu Zhao et al.
Runtime repair layer that abstracts agent runs into a dependency-aware critical-transition graph, localizes failure-critical subtrajectories after a run, and guides recovery on re-execution without modifying the underlying agent. Research preprint
📄 SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review 2026 · arXiv
Ruoyu Wang et al.
Replaces one-shot PR generation with a generate-review-revise loop in which a reviewer agent explores the repository, accepts or rejects the PR, and feeds structured feedback into revision, with an accompanying benchmark and trajectory dataset. Research preprint
📄 Reason Less, Verify More: Deterministic Gates Recover a Silent Policy-Violation Failure Mode 2026 · arXiv
Vikas Reddy et al.
Finds that 78% of observed agent failures in a tau^2-bench domain are silent wrong-state failures invisible to both the tool and the agent's self-report, and that deterministic read-only pre-execution gates in the loop recover them. Research preprint
📄 Harnessing Code Agents for Automatic Software Verification 2026 · arXiv
Shuangxiang Kan et al.
Wraps a general code agent in a verification harness and lets it run until every targeted Coq lemma is proved, beating fixed human-designed proof strategies and reaching full lemma coverage with no expert intervention. Research preprint
📄 LLM-as-a-Verifier: A General-Purpose Verification Framework 2026 · arXiv
Jacky Kwok et al.
Treats verification as a scaling axis and builds a training-free framework that computes continuous scores from token logits for fine-grained agentic feedback, scaled via score granularity, repeated evaluation, and criteria decomposition. Research preprint
📄 From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents 2026 · arXiv
Joongho Ahn & Moonsoo Kim
Moves deterministic agent behavior out of prompts into code, schemas, and behavior contracts, wrapping validation around a replaceable model boundary so enterprise agents remain auditable and safe across model substitutions. Research preprint
📄 From Noisy Traces to Root Causes: Structural Trajectory Analysis and Causal Extraction for Agent Optimization 2026 · arXiv
Ying Chang et al.
STRACE structures redundant, heterogeneous agent execution traces by mining batch-level failure patterns and performing causal localization over a textual dependency graph, handing root causes rather than noisy trajectories to the reflection-based optimizer and lifting success on a formal verification task from 42.5% to 58.5%. Research preprint
📄 Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems 2026 · Conference on Language Modeling (COLM)
Yufei Xia et al.
AgentLocate attributes failures in LLM multi-agent trajectories to both the responsible agent and the earliest decisive step, pairing LLM-based evaluation with independent assessor verification and confidence-weighted aggregation to outperform prior attribution methods on two benchmarks, the diagnose side of the verify step for dispatched-agent loops. Research paper
📄 3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse 2026 · arXiv
Shyam Agarwal et al.
Builds a causal theory of 26 constructs and 67 relationships from 3,100 coded practitioner documents on how AI-authored pull requests reshape code review, arguing review is the control point through which a coding agent's effect on software is decided and that outcomes hinge on team expertise and review process structure rather than AI itself. Research preprint
📄 Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring 2026 · arXiv
Jennifer Za et al.
Stress-tests chain-of-thought monitoring as an in-loop safety gate: adversarial agents arguing for policy-violating proposals turn the scratchpad into a persuasion channel, with monitor access to the agent's reasoning increasing approval of harmful actions by 9.5% on average, while pairing monitor and fact-checker from different model families cuts violating approvals by up to 45%. Research preprint
📄 Physics-Audited Agentic Discovery in Scientific Machine Learning 2026 · arXiv
Diab W. Abueidda et al.
Verification-first workflow (PA-SciML) for agentic model discovery in scientific ML: fixes the scoring evaluator before search, derives machine-checkable physics requirements (boundary conditions, superposition, stiffness scaling, causality), audits every trained candidate's predicted fields against them, and separately searches prescribed input ranges for high-violation cases, reporting a surrogate as verified only under the stated checks; a domain-specific case of verification-gated agentic search. Research preprint
📄 Bug Report Specification Refinement with Trajectory Guidance for Automated Program Repair 2026 · arXiv
S M Farah Al Fahim et al.
TrajSpec runs a trajectory-collection agent over the pre-fix repository and mines the unverified trajectory for specification evidence, refining vague bug reports into structured specifications that guide automated program-repair loops. Research preprint
📄 Failure as a Process: An Anatomy of CLI Coding Agent Trajectories 2026 · arXiv
Xiangxin Zhao et al.
Empirical anatomy of 3,843 CLI coding-agent trajectories across seven models and three scaffolds (OpenHands, MiniSWE, Terminus2), with 1,794 fully annotated over 63,000+ manually reviewed steps; models failure as a temporal process of onset, evolution, and recovery and finds failures dominated by epistemic errors that begin within the first few steps yet stay undetected until recovery is impossible, arguing for in-loop validation and intervention over final-outcome evaluation. Research preprint
📄 Agentic Proof and Property-Based Testing via Property-Templates 2026 · arXiv
Seongmin Lee et al.
Dual-track verification-in-the-loop for AI-generated code: shared property templates drive both formal proof in Lean 4 and executable property-based tests for PySpark, raising agentic proof success up to 2.6x and cutting proof hallucinations by 59%. Research preprint
📄 AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP 2026 · arXiv
Aritra Mazumder & Nusrat jahan Lia
Workbench that reproduces an agent failure, intervenes at the point it went wrong, and tests mitigations, turning one-off agent bugs into a repeatable diagnose-and-fix loop over MCP tool use. Research preprint
📄 Latent Programming Horizons in Coding Agents 2026 · arXiv
André Silva et al.
Shows a coding agent's hidden states linearly encode program properties like correctness and test outcomes and predict future edits up to 25 steps ahead, a latent signal that could gate or steer verification loops before edits materialize. Research preprint
📚 Why evaluate agents Google Agent Development Kit Official ADK guide to evaluating final responses and trajectories, defining test cases, selecting criteria, and running repeatable agent evaluations locally or in CI. Official documentation
📄 Structured Feedback Improves Repair in an LLM Agent Loop 2026 · arXiv
Jaideep Ray & Ankit Goyal
In 50 paired TextWorld tasks under a four-call budget, feedback containing the failure location, observed value, and admissible alternatives raises repair success from 14/50 to 36/50 for one model and 8/50 to 29/50 for another; ablations identify alternatives, not JSON syntax, as the main driver. Research preprint
📄 Copy-on-Write Scoring: Application-Specific Agent Evaluations 2026 · ICML Workshop on Agents in the Wild: Safety Security and Beyond
Joanna Roy & Sven Hoelzel
Uses PostgreSQL copy-on-write isolation to let an agent modify a realistic application state while a scorer evaluates the resulting operations safely; the Plane case study also exposes tool-surface defects that simpler task checks miss. Research paper
📄 The Prover Is the Judge: Verified Security Software from AI Coding Agents in Ada/SPARK 2026 · arXiv
Tobias Philipp
Places formal proof obligations inside a coding-agent repair loop and reports 49,280 discharged obligations with 20-40x less supervision; the paper also states that proofs must be paired with known-answer tests, interoperability checks, and human specification review. Research preprint
📄 Verified LLM-Driven Synthesis for Concept Design 2026 · arXiv
Alcino Cunha
Combines formal concept-and-reaction semantics with a counterexample-guided LLM synthesis loop whose candidates must satisfy machine-checked invariants; experiments on three applications show scenarios steer the loop more consistently than natural-language prompts, while sparse scenarios overfit and nondeterministic omissions still limit coverage. Research preprint
📄 AEGIS: Assay-Aware Protocol Validation and Runtime Monitoring for Open-Source Liquid Handling Robots 2026 · arXiv
Priyanka V. Setty et al.
Pairs preflight LLM checks against machine-readable assay rules with runtime visual monitoring of physical execution; reports adjusted F1 0.97 for protocol validation and average precision 0.89 for trajectory monitoring, while explicitly surfacing the small-pipette and transparent-liquid limits. Research preprint
Resource Published at Contribution Evidence
📄 GLEAN: Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verification 2026 · arXiv
Yichi Zhang et al.
Compiles expert guidelines into step-wise trajectory checks, calibrates accumulated evidence into correctness probabilities, and triggers additional verification when uncertainty remains high. Research preprint
📄 Zombie Agents: Detecting Semantic Livelock in Long-Horizon Autonomous Software 2026 · Proceedings of the 3rd ACM International Conference on AI-Powered Software (AIware '26)
Simarjot Khanna
Defines semantic livelock as continued agent activity without progress and proposes an independent embedding-based convergence monitor that detected the pattern in 25% of the analyzed long-duration SWE-agent failures. Research paper ‡
📄 Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents 2026 · arXiv
Xing Zhang et al.
Every self-improvement loop rests on an evaluation metric that can itself drift or be gamed; the Double Ratchet framework co-evolves metrics alongside agent skills with anchor discipline and independent audits, retaining 88-110% of ground-truth lift across code generation, SQL, and report-writing loops. Research preprint
📄 AgentLTL: A Trace-Verification Framework for Measuring, Enforcing, and Training Procedural Compliance in Tool-Using LLM Agents 2026 · arXiv
Laïla Elkoussy & Julien Perez
First-Order Linear Temporal Logic specification language for verifying that tool-using agents follow procedural rules across the whole trace, not just reach correct answers; supports real-time in-loop tool-call gating and dense-reward training, with compliance gains that generalize to unseen procedural variations. Research preprint
📄 Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade 2026 · arXiv
Kai Ruan et al.
Trains lightweight linear probes on hidden states from an episode's earliest interactions to predict eventual failure, then aborts doomed runs through a calibrated cascade with user-specified recall guarantees, cutting generated tokens by roughly 54-60% at a 90% recall target on TextCraft and WebShop across three models, a when-to-stop gate that stops wasted runs instead of letting them burn to timeout. Research preprint
📄 Where Does Agent Reliability Come From? A Cross-Benchmark Decomposition of Verification Loops, Specialist Models, and Scaffolding in a Production Enterprise Agent 2026 · arXiv
Arunabh Dastidar
Ablates a production enterprise agent (Leni) across three benchmarks to decompose reliability gains among verification loops, specialist models, and scaffolding: 7-15pp improvements come mostly from scaffolding, routing, and specialist models, while the verification step's isolated gain is small (+1.5pp) yet rescues otherwise-failing tasks, and swapping the small trained verifier for the generating frontier model eliminates most rescues. Research preprint
📄 Rethinking Verification in Agent Harnesses 2026 · arXiv
Zhengxin Zhang et al.
Of 17 verification designs on seven frontier models, a fresh-context critic added 5.9-9.5 Terminal-Bench points at 1.13-1.25x cost; reject-completion gates cost 1.60-1.76x. Exit checks belong with a separate critic. Research preprint
📄 Test Coverage Analysis of Agentic Pull Requests 2026 · arXiv
Atish Kumar Dipongkor et al.
Mines 4,882 agent-generated PRs (five coding agents) from the AIDev dataset and finds agents ship tests in only 49.6% of code-changing PRs, existing tests cover just 27% of changed Python lines, and error-handling code goes unexecuted at up to 86% miss rates, empirical evidence that unattended agent loops under-verify their own output (to appear at ICSME 2026). Research preprint
📄 TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimization 2026 · arXiv
Alex Mathai et al.
Attributes verbose "CodeSlop" to the agent loop's own search process, speculative edits, abandoned hypotheses, and temporary changes that survive into the final diff, and removes 17.9-32.9% of redundant code across agentic scaffolds by minimizing the trajectory rather than post-editing the code, with negligible performance regression (Columbia and Google). Research preprint
📄 Agentic Code Review in the Terminal: A Trajectory-Level Analysis of Behavior, Cost, and Human-Alignment 2026 · arXiv
Wachiraphan Charoenwet et al.
Trajectory-level study of terminal-based code-review agents that give feedback before pull-request creation, finding they achieve higher review precision but incur substantial exploration and validation overhead, and that successful reviews are associated with stronger planning and less downstream validation, behavior and cost evidence for designing review-agent gates. Research preprint
🧰 brain0 2026 · GitHub
Brain0-ai/brain0
Black-box decision graph for AI-written code that records why an agent made each change, making agent-authored diffs inspectable after the fact. Source implementation
🧰 Watch Skill 2026 · GitHub
oxbshw/watch-skill
Video understanding and self-verification layer that turns videos, streams, and agent screen recordings into searchable evidence an agent can check its own work against. Source implementation
🧰 Auto 2026 · GitHub
RightNow-AI/auto
Compiles recorded LLM-agent behavior into verified, capability-confined WebAssembly binaries, proving which parts of a loop are secretly symbolic and distilling them into deterministic code. Source implementation
📄 Critic Experience Bank: Self-Evolving Step-Level Confidence Estimation for LLM Agents 2026 · arXiv
Yaopei Zeng et al.
Training-free verification-in-the-loop: a hindsight reviewer labels which steps of completed trajectories were genuinely productive, building an experience bank that calibrates the critic's step-level confidence on future runs, improving when a loop should trust, retry, or escalate an action. Research preprint
📄 Tracing Agentic Failure from the Flow of Success 2026 · arXiv
Samuel Yeh et al.
Failure attribution for agent loops that learns only from successful trajectories: OAT uses neural controlled differential equations to spot the error step in failed runs, 200-5000x faster than prompting-based attribution with ~20% better F1, needing just 100 success traces, makes in-loop failure localization cheap enough to run continuously. Research preprint
📄 Beyond Test Presence: Assessing the Quality and Robustness of Agent-Generated Tests 2026 · arXiv
Preet Jhanglani et al.
Study of 204,673 test artifacts comparing agent-generated and human tests: agents win on edge-case and boundary coverage but produce flakier tests (file I/O, non-determinism), a caution that the verification step of a loop can itself become the unreliable component if agent-written checks go unaudited. Research preprint
📝 What a Verification Loop Adds to a Coding Agent: A First Look Medium First-hand write-up measuring what wrapping a coding agent in an explicit verification loop changes in practice, comparing gated and ungated runs on the same tasks. Practitioner analysis ‡
📄 Best-of-Evidence: Best-of-N Selection under Partial Verification 2026 · arXiv
Cenwei Zhang et al.
Inference-time selection framework replacing whole-response Best-of-N scoring with budgeted claim-level partial verification, using a signed candidate-factor graph so one checkable finding can support part of the candidate pool while contradicting the rest, with proven logarithmic-vs-linear query savings from shared evidence. Research preprint
📄 Toward Continuous Assurance for the Democratization of AI Agent Creation in Industry 2026 · arXiv
Natan Levy & Harel Berger
Position paper (Levy & Berger, Jul 2026) on the silent-degradation gap of low-code/no-code agents whose dependencies, models, tools, retrieval sources, permissions, schedules, drift after deployment; proposes continuous assurance via dependency mapping, readiness contracts, scheduled checks, and lifecycle governance, with an initial auditor prototype. Research preprint
📚 Catch Security Issues as Claude Writes Code Claude Code Docs Official docs for Anthropic's security-guidance plugin (July 2026), which wires a verification loop into Claude Code's own lifecycle entirely via hooks: a deterministic per-edit pattern check with no model call, a background fresh-context model review of each turn's Git diff that re-prompts Claude with its findings, and a deeper agentic review on every commit or push that reads surrounding code, a worked example of layering independent, non-blocking verification gates inside the agent loop. Official documentation
📄 pAI-Econ-claude: A Gated Human-in-the-Loop Multi-Agent Architecture for AI-Assisted Economic Theory Development 2026 · arXiv
Chen Zhu et al.
Gated human-in-the-loop multi-agent architecture for domains lacking any cheap machine-checkable correctness signal, using economic theory development as the case study. Research preprint
🧰 Review Loop 2026 · GitHub
earendil-works/pi-review-loop
First-party extension from earendil-works, the org behind the pi coding agent (77k stars): keeps a native review window open while the agent works, and submitting a review records the current workspace as a session-backed checkpoint so the next pass diffs only the increment ("Since review" vs "vs HEAD" modes), turning one-shot PR review into a persistent human-verification loop over continuous agent output. Source implementation
📄 Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair 2026 · arXiv
Xueping Gao et al.
Directly attacks the core Loop Engineering assumption that more generate-test-revise iterations equal more reliability. Over 900 three-revision trajectories on HumanEval repairs, current correctness DROPS from 0.820 after one revision to 0.673 after two even as ever-correct rises to 0.847 -- the loop finds the patch and then loses it. Research preprint
📄 Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents 2026 · arXiv
Diandian Guo et al.
Names and measures the verifier-deployment gap: when a self-improving agent controls both the artifact it rewrites and the tests that judge the rewrite, self-assigned scores stay near-perfect while sealed deployment performance degrades. Introduces a Sealed Exogenous evaluation protocol and asks the practical question -- how little exogenous trust suffices to stop real regressions from shipping. Research preprint
📄 Falsifiable Commitment Planning for Self-Correcting Web Agents 2026 · arXiv
Guangyi Liu et al.
Long-horizon web agents drift: a trajectory stays locally plausible long after the state or plan assumption stopped supporting the instruction. Research preprint
📄 Adversarial Test-Hardening for AI-Written Code: An Instrument Autopsy and a Pre-Registered Causal Estimate of the Critic Loop 2026 · arXiv
Jeff Otterson
A Tester model writes tests, mutation testing names surviving injected defects, and a Critic model writes tests to kill exactly those -- every verdict decided mechanically so no model judges another. The loop killed 105 mutants one-shot generation missed and lost none. Research preprint
📄 Beyond Aggregate Risk: Role-Stratified Conformal Risk Control for LLM Tool Calls 2026 · arXiv
Md Ashikur Rahman et al.
Tool-call arguments carry wildly different risk -- untrusted content may safely shape an email body but must never determine a recipient, account, command, or credential -- yet standard statistical control certifies the action as a whole, letting rare high-risk field failures hide behind benign arguments. Research preprint
📄 From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement 2026 · arXiv
Qinsi Wang et al.
RLVR works where correctness is deterministically checkable and stalls everywhere else, leaving open-ended tasks dependent on preference data, reward models, or LLM judges with their bias, capability ceilings, and inference cost. Research preprint
📄 A Frozen 12B Beats Frontier Models on Verified Work: 100% Accuracy, 0 Tokens, Bit-Exact, Forever 2026 · arXiv
Sietse Schelpe
Verification-gated memoization taken to its limit: the model stays frozen while a persistent store accumulates only those solutions that passed an independent verification step, so later encounters with a seen task replay a checked answer instead of re-deriving it. Read the headline accuracy and token figures as properties of cache hits on verified work, not as general model capability. Research preprint
📄 CodeSpec: Dual Executable Specifications for Agentic Long-Horizon Feature Development 2026 · arXiv
Peiding Wang et al.
Pairs architecture and behavior specifications as executable artifacts so a code agent can verify design completeness and hold design-implementation consistency across a long repository-level feature build, beating Claude Code baselines. Makes the spec a runnable gate rather than a prompt preamble. Research preprint
📄 SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch 2026 · arXiv
Yihao Chen et al.
Splits from-scratch program synthesis into a behavioral-specification elicitation stage and a code synthesis stage, improving test pass rates and exploration coverage across models and benchmarks. Useful evidence that the specification step deserves its own loop iteration instead of being folded into generation. Research preprint
📄 SARC-DQ: Runtime Data-Quality Gating for Agentic AI: Silent Evidence Defects, the Incompetence Shield, and Downstream-Only Remediation 2026 · arXiv
Gaston Besanson
Shows a competent agent silently converts an injected metadata-borne defect (e.g. a stale price) into a costly action about 60% of the time, and proposes a metadata-aware pre-action gate plus downstream remediation. Names a failure mode, the defect is invisible to the agent, so more capability does not help, that pure output-checking gates cannot catch. Research preprint
📄 Before Agents Speak: Pre-hoc Failure Risk Inference in Multi-Agent Systems 2026 · arXiv
Shi Lin et al.
HalluProp estimates individual agent failure and emergent system-level hallucination risk before inter-agent interaction begins, by scoring agent-task semantic alignment and modeling propagation across the communication network. Shifts multi-agent verification from post-hoc detection to admission control. Research preprint
📄 ARCHER: Agentic Rule and Compliance Harness for Executable Regulations 2026 · arXiv
Chiraag Singh Anand et al.
Multi-agent harness that compiles natural-language regulations into auditable verification code, making compliance checks transparent and rerunnable, with open models matching frontier APIs at lower cost. The regulation-to-executable-check pipeline generalizes well beyond its building-code case study. Research preprint
📄 F(AI)2R: Who Did What, and Who Checked? Verifiable AI Provenance as an Executable Skill 2026 · arXiv
Florian Krebs
Packages provenance capture as an executable agent skill (aiprov) that records activities, claims, and sources during production while reserving verification authority for humans. Answers the audit question every long-running agent loop eventually faces: who checked this, and can you prove it. Research preprint
🧰 Codex Security 2026 · GitHub
openai/codex-security
OpenAI's Codex Security CLI and TypeScript library, giving a coding agent a first-party path to run security review over its own changes instead of depending on an external scanner bolted onto the loop. Source implementation
📚 Copilot Code Review: Agent Skills and MCP Now Generally Available 2026 · The GitHub Blog GitHub changelog, 2026-07-29: the automated review gate becomes programmable and context-aware for all Copilot Pro, Pro+, Business, and Enterprise users. Technical documentation
🧰 old-coder 2026 · GitHub
AmazingAng/old-coder
Markdown skill that makes a coding agent prove its work instead of asking you to read every line: the agent pushes each change through a spec, then a gauntlet, then evidence. Plain instructions rather than a runtime, so it works with Claude Code, Codex CLI, Cursor, Aider, or a custom loop. Source implementation
📄 Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug? 2026 · arXiv
Xiaonan Xu & Wenjing Wu
Attacks the assumption every verification gate rests on: that a passing test is evidence about the reported defect. Research preprint
📄 Preventing Premature Commitment in Coding Agents with an Evidence-Conditioned Execution Layer 2026 · arXiv
Yisen Xu et al.
Names premature commitment as a distinct failure pattern: coding agents edit source or submit patches before examining enough repository evidence to justify the change. Research preprint
📝 Ten advances in mathematics and theoretical computer science OpenAI OpenAI's Aug 1 writeup of ten previously-open problems in high-dimensional geometry, coding theory, group theory, operator algebras, arithmetic circuit complexity, quantum complexity, lattices and extremal combinatorics, closed by an internal Astra model. Practitioner analysis
📝 Postmortem for Kernel Soundness Bug #14576 2026 · leodemoura.github.io
Leonardo de Moura
Leonardo de Moura, Lean's creator, dissects a kernel soundness bug surfaced when Ramana Kumar published an AI-assisted "disproof" of the Collatz conjecture on Jul 25; Kiran Gopinathan reduced it to a minimal exploit on Jul 28 and it was patched within an hour. Practitioner analysis
📄 When AUC 0.998 Is Not Enough: A Candidate Evaluation Protocol for Hidden-State Probes of Indirect Prompt Injection in Multimodal Computer-Use Agents 2026 · arXiv
Yanhang Li et al.
A methodological negative result aimed squarely at teams building detector-based verification gates. The claim: a high probing AUC on a clean-vs-attack split is not, on its own, evidence that the probe detects malicious content, it may simply be recognizing surface artifacts of how the attack was constructed. Research preprint
📄 VeriSkill: A Self-Evolution Framework for Program Verification Skills 2026 · arXiv
Changguo Jia et al.
Closes the loop between a verifier's rejection and the agent's skill library: verification failures are attributed to specific skill deficiencies and distilled into reusable lessons, so the next attempt is informed by a diagnosis rather than a raw error dump. Research preprint
📄 LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger 2026 · arXiv
Enjun Du et al.
Models the agent trajectory as a provenance-constrained state machine: tool outputs normalize into a ledger and every reasoning claim must cite a ledger entry, verified at entity and numeric level. Research preprint
📄 Leveraging Trajectory Graphs for Pre-Execution Error Diagnosis in Agentic LLM Systems 2026 · arXiv
Xu Zheng et al.
Shifts the gate earlier than the usual post-hoc check: a GNN trained on historical trajectory graphs flags action sequences likely to fail before they execute, letting the agent self-correct pre-emptively. A cheap pre-execution veto is exactly the primitive unattended loops need to avoid burning steps on doomed branches. Research preprint
📄 OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models 2026 · arXiv
Qiushi Sun et al.
Benchmarks the judge rather than the agent, showing that even frontier VLM judges share a systematic leniency bias that marks failed computer-use runs as successes, the failure mode that quietly corrupts any loop gated on an LLM verifier. Ships OS-Shepherd (9B/35B) open reward models matching commercial judges at 30-60% lower cost. Research preprint
📄 Agentic Method for Deterministic Validation of Legacy Code Migration 2026 · arXiv
Andras Ferenczi et al.
The 'Locksmith Loop' instruments both source and target programs and iteratively searches for inputs that penetrate unexplored branches, identifying what blocks coverage rather than accepting an LLM's claim of equivalence. Research preprint
📄 Safety-Gated Agentic Supervisory Control on a Coupled Distillation Benchmark: Regime Map, Auditable Gate, and Co-Design Findings 2026 · arXiv
Christian Rosenthal
A rule-based auditable gate sits between an LLM supervisory controller and the regulatory layer, blocking unsafe setpoints. Research preprint
📄 Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correction 2026 · arXiv
Pan Wang et al.
Coding agents get typed recovery signals from compilers and tests, but general agent tasks expose only a coarse failure, so generic recovery playbooks widen context exactly when a narrower repair is needed. DARC profiles failure modes per task family in advance and prunes the interventions that cannot apply. Research preprint
📄 From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents 2026 · arXiv
Yiqi Wang et al.
Survey arguing final-answer accuracy cannot say which evidence supported a claim, whether a tool call was justified, or where a failure began. Defines execution provenance as the typed graph of a run and evidence tracing as its projection onto support relations, then taxonomizes trace sources and granularity. Research preprint
📄 LongHorizon-Harness: Advancing Long-Horizon Agents for Real-World Tasks 2026 · arXiv
Ziyu Ma et al.
Reformulates long-horizon execution as task-state management, on the observation that holding state and completion assessment inside a growing context lets a wrong self-assessment propagate into later decisions. State lives outside execution and updates only on facts an independent read-only auditor verified from the environment. Research preprint
📄 GameGen-Verifier: Parallel Keypoint-Based Verification for LLM-Generated Games via Runtime State Injection 2026 · arXiv
Chaobo Jia et al.
Correctness here is defined over long-horizon interaction, so a generated game can look right while breaking state updates, interaction rules, or phase transitions. Argues agent-as-verifier collapses into open-ended play, leaving verdicts reachability-bound, slow, and dependent on the verifier's own skill. Research preprint
📄 Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failures 2026 · arXiv
Isham Kalappurackal Mansoor et al.
Agent frameworks usually treat a tool call as atomic with a binary result, while real systems time out after dispatch, update state partially, and make effects visible late. Wraps calls with postcondition verification, verify-before-retry, and idempotency keys so a retry cannot duplicate an action that already landed. Research preprint
📄 RECEIPT: Measuring and Repairing the Return Assumption of LLM Agents in Web Service Operations 2026 · arXiv
Boyang Yang et al.
Without a terminal, 7 of 8 tested harnesses close command input, and 11.5% of prompting CLI calls then exit 0 without acting; reporting process state cut false success claims by 73.5%. Check effects, not exit codes. Research preprint
📄 Don't Blindly Trust It: How Unreliable Feedback Breaks Tool-Using LLM Agents 2026 · arXiv
Chubin Zhang et al.
Fixes the loop, prompt, action space, and decoding, then varies only whether the returned observation is faithful, misleading, or absent. Persistent misleading feedback inverts the value of the tool: on HotpotQA one model scores 44.8 F1 with clean retrieval, 22.3 with no feedback at all, and 4.7 under shuffled retrieval. Research preprint
📄 Human-on-the-Loop Orchestration for AI-Assisted Legal Discovery 2026 · arXiv
Anushree Sinha et al.
Names trajectory collapse: an early misclassification propagates silently and invalidates an entire privilege review, which in e-discovery can amount to malpractice. Answers with verification layered across planning, reasoning, execution, and uncertainty, plus mandatory escalation that suspends autonomous action. Research preprint
📄 Affordance Agent Harness: Verification-Gated Skill Orchestration 2026 · arXiv
Haojian Huang et al.
Closed-loop skill orchestration for grounding, against fixed pipelines that ignore per-instance difficulty and cannot recover from an intermediate error. The loop has to decide whether the evidence it has gathered is reliable enough to commit, under bounded inference cost and without labels. Research preprint
📄 The Reach of a Verification Tool Decides Its Value: A Controlled Study of Verification Surface, Artifact Quality, and Cost in AI Coding Agents 2026 · arXiv
Achint Mehta
Holds everything fixed except the agent's verification surface, the tools it can use to check its own work, and builds 1,116 web applications across six models and eight tool configurations graded condition-blind against a frozen rubric. Tests whether a wider surface actually buys matching quality. Research preprint
📄 Who Verifies the Builder? Toward Independent Assurance for Agent-Produced Software 2026 · arXiv
Sion Israel Sion & Alain April
When one agent writes both the code and its tests, a passing suite stops being independent evidence. Separates verification from building with a spec-first phase that never sees the implementation; a fresh-context verifier caught more of 46 seeded faults. Research preprint
📄 Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone 2026 · arXiv
Alejandro Cuadron & Arpit Gupta
Ten coding agents complete 95% of requests alone but 51% of 280 twenty-request chains, because self-tests miss mistakes later requests build on; one external 'that's wrong' and a repair round lift chains to 59%. Research preprint

Securing Unattended Loops

A loop that runs while nobody watches needs stronger boundaries than an interactive session. These resources cover the main risks: untrusted intake content, over-broad permissions, and unsandboxed execution.

Resource Published at Contribution Evidence
⚠️ The lethal trifecta for AI agents 2025 · Simon Willison’s Weblog
Simon Willison
Simon Willison's rule of thumb: private data, untrusted content, and an exfiltration channel must never meet inside one unattended agent. Risk analysis
⚠️ Prompt injection series Simon Willison’s Weblog
Simon Willison
Ongoing series on the core unsolved vulnerability for loops whose intake includes content written by strangers. Risk analysis
📚 Agentic AI - Threats and Mitigations genai.owasp.org OWASP threat model for agentic systems, useful when reviewing intake, memory, tool, and delegation boundaries. Technical documentation ‡
📚 Designing AI agents to resist prompt injection OpenAI OpenAI's official defense-in-depth guidance: least privilege, sandboxed tools, output verification, and human confirmation for the high-impact actions an unattended loop might take. Technical documentation
🧰 sandbox-runtime 2025 · GitHub
anthropics/sandbox-runtime
Anthropic's OS-level filesystem and network sandboxing for arbitrary processes without requiring a container. Source implementation
🧰 E2B 2023 · GitHub
e2b-dev/E2B
Open-source isolated cloud sandboxes for running untrusted, AI-generated code inside agent loops. Source implementation
📚 Modal Sandboxes Modal Secure sandboxed execution for agent-driven code with resource limits and network controls. Technical documentation
🧰 Daytona daytona.io Infrastructure for running AI-generated code in fast, isolated sandboxes. Implementation
🧰 peerd 2026 · GitHub
NotASithLord/peerd
Browser-extension harness that runs the agent loop entirely client-side with user-supplied keys, sandboxed compute, and per-environment actor agents that hold only their tools and no API keys, isolating the orchestrator from untrusted content as a prompt-injection boundary. Source implementation
📄 When Claws Remember but Do Not Tell: Stealthy Memory Injection in Persistent Personal Agents 2026 · arXiv
Yechao Zhang et al.
Shows one poisoned email can write hidden entries into a persistent personal agent's long-term memory that silently alter future unattended runs, introducing the 108-case WhisperBench evaluation and the MemGhost attack that reaches 87.5% success. Research preprint
📄 Your Agent's Memories Are Not Its Own: Forged Reasoning Attacks on LLM Agent Memory and Defenses 2026 · arXiv
Neeraj Karamchandani et al.
Introduces FARMA, an attack that plants forged reasoning traces in an agent's persistent memory so poisoned rationales carry into future runs, and SENTINEL, a reasoning-guard defense that cut attack success from up to 100% to zero in evaluation. Research preprint
📄 Distributed Attacks in Persistent-State AI Control 2026 · arXiv
Josh Hills et al.
Extends AI-control evaluation to coding agents shipping code that persists across sessions, showing a misaligned agent can spread an attack across successive PRs to evade per-transcript monitors, and adds a stateful link-tracker monitor that cuts evasion from 93% to 47%. Research preprint
📄 ElephantAgent: Contextual State Continuity in Agentic Systems 2026 · arXiv
Jiankai Jin et al.
Verification protocol that recomputes state digests before each query and logs authorized changes to a trusted-hardware ledger, so an agent's persistent memory and tool descriptions cannot be covertly poisoned between runs and can be rolled back to the last verified state. Research preprint
🧰 Cloudflare security-audit-skill 2026 · GitHub
Cloudflare/security-audit-skill
Cloudflare's open-sourced six-phase audit pipeline in which separate validation agents try to disprove each finding and fresh agents independently verify every claim against source code, emitting schema-validated findings that accumulate across repeated runs. Source implementation
📄 The Balkanization of Execution-Security Research for AI Coding Agents 2026 · arXiv
Mohammadreza Rashidi
Systematizes 39 papers on the execution layer around coding agents, spanning sandbox isolation, capability control, TOCTOU races, MCP threats, and egress control, surfacing five cross-cutting gaps and four verified CVEs in production agent harnesses. Research preprint
📄 Context-to-Execution Integrity for LLM Agents 2026 · arXiv
Igor Santos-Grueiro
Execution-boundary system where a deterministic gate admits a tool call only after field authority, exact-effect authorization, and invocation authority all bind to the same action manifest, protecting loops that read attacker-writable context. Research preprint
📄 When Agents Remember Too Much: Memory Poisoning Attacks on Large Language Model Agents 2026 · arXiv
George Torres et al.
GhostWriter is a two-phase attack that poisons the long-term memory store of tool-using personal agents so injected content persists across runs and activates in later tasks. Research preprint
📄 Token-Flow Firewall: Semantic Runtime Auditing for Persistent AI Agents 2026 · arXiv
Puji Wang et al.
Proposes TokenWall, a runtime firewall that audits a long-lived agent's semantic flows (memory updates, tool arguments, inter-component messages) before they reach privileged sinks, reporting attack success reduced to 12.5% with a 97.4% benign pass rate and 0.69s added latency. Research preprint
📄 Prismata: Confining Cross-Site Prompt Injection in Web Agents 2026 · arXiv
Corban Villa et al.
Applies contextual least privilege to web agents by dynamically labeling page content with trust levels and mechanically confining what the agent can see and do, cutting cross-site prompt-injection attack success on benign pages without requiring developer annotations. Research preprint
📄 TRACE: A Two-Channel Robust Attribution Watermark via Complementary Embeddings for LLM-Agent Trajectories 2026 · arXiv
Zheng Gao et al.
Embeds a two-channel attribution watermark in LLM-agent trajectory logs (one channel keyed on content for deletion resistance, one on log structure for rewrite resistance) so provenance survives an adversary with full read/write access, reporting detection scores near z = 100 on long-horizon ToolBench and ALFWorld trajectories with no loss of agent performance. Research preprint
📄 Beyond Attack-Success Rate: Action-Graded Severity Scale for Tool-Using AI Agents 2026 · arXiv
Harry Owiredu-Ashley
Replaces binary attack-success red-teaming metrics with a seven-level ordinal severity rubric (L0-L6) that grades harm along the agent's tool-call trajectory by action reversibility, scope expansion, and privilege escalation, validated with deterministic analysis and frontier-model judges across multiple victim models and defenses. Research preprint
📄 Beware of Agentic Botnets: Scalable Untargeted Promptware Attacks via Universal and Transferable Adversarial HalluSquatting 2026 · arXiv
Aya Spira et al.
Introduces adversarial hallucination squatting, in which attackers pre-register resource names LLMs predictably hallucinate (at rates up to 85-100%) and plant universal, cross-model-transferable promptware payloads on the open web, reaching agent loops that autonomously ingest internet content with no direct injection channel. Research preprint
📝 GitLost: How We Tricked GitHub's AI Agent into Leaking Private Repos noma.security Noma Labs researcher Sasi Levi shows how hidden plain-English instructions in a malicious GitHub Issue make an issue-triggered GitHub Agentic Workflows agent exfiltrate private-repo contents into public comments, bypassing GitHub's data-leak guardrails with a one-word reframe, disclosed responsibly to GitHub. Practitioner analysis
🧪 ScopeJudge: Cost-Aware Pre-Execution Gating for Offensive Security Agents 2026 · arXiv
Shane Caldwell et al.
Benchmark of 4,897 tool calls from offensive-security agent trajectories, labeled by professional penetration testers, for studying pre-execution gating where a cheap trusted LLM judge accepts or rejects each proposed call before it runs and the engagement boundary must be inferred from the request rather than a fixed policy. Benchmark or evaluation
📄 Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors 2026 · arXiv
Oliver Makins et al.
Studies AI control when several agents operating jointly on shared infrastructure pursue a malicious goal, evaluated on FakeLab (synthetic AI-lab codebase: 9 services, 86 benign tasks, 4 attack scenarios); finds a fragmentation effect (the more agents coordinate an attack, the less likely per-agent monitors catch any single attacker) and that explicit planners amplify attack success, directly relevant to monitoring fleets of background agents. Research preprint
📄 Mitigating Taint-Style Vulnerabilities in MCP Servers via Security-Aware Tool Descriptions 2026 · arXiv
Yang Shi et al.
Finds taint-style flaws make up a substantial fraction of MCP-server vulnerabilities and normally demand context-specific code fixes, then proposes SPELLSMITH, which hardens tool descriptions with security-aware behavioral guidance so the agent's own self-reflection steers it away from triggering the vulnerable flows, mitigating multiple vulnerability classes at the tool-protocol layer without code-level patches. Research preprint
📝 Factory Droid Shield 2.0: Learned Secret Detection for Autonomous Commits 2026 · Factory
Factory
Factory upgrades the verification gate on every autonomous Droid commit with a two-model pipeline flanking the deterministic secret scanner, pairing a high-recall risk model with a precision referee. Practitioner analysis
🧰 destructive_command_guard 2026 · GitHub
Dicklesworthstone/destructive_command_guard
Rust safety hook that intercepts and blocks destructive Git and shell commands (hard resets, recursive deletes, database drops) before AI coding agents execute them, across Claude Code and other harnesses. Source implementation
📄 Friendly Fire: Hijacking Defensive Cyber AI Agents for Remote Code Execution 2026 · AI Now Institute
Boyan Milanov
Proof-of-concept showing prompt injections spread across ordinary repository files can hijack defensive security agents into remote code execution, demonstrating that even security-focused agent loops inherit the untrusted-content attack surface. Research paper
📝 How We Contain Claude Across Products 2026 · Anthropic Anthropic engineering on capping agent blast radius with three containment architectures matched to threat models, including ephemeral sandbox containers for untrusted code execution. Practitioner analysis
📄 Rethinking MCP Security: A Large-Scale Study of Runtime MCP Servers and Scanner Reliability 2026 · arXiv
Pei Chen et al.
Large-scale study of live MCP servers finding widespread security weaknesses and that existing MCP security scanners miss or misreport many of them, a gap for anyone gating agent tool access on scanner output. Research preprint
📄 Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming 2026 · arXiv
Xutao Mao et al.
Uses an autonomous research loop to red-team production agents, having one agent iteratively discover, reproduce, and refine attacks against another, turning red-teaming itself into a recurring verified loop. Research preprint
📄 Temporary Authority, Permanent Effects: Commit-Time Authorization for LLM Agents 2026 · arXiv
Igor Santos-Grueiro
Proposes binding an agent's authority to the moment of commit rather than the moment of request, so a permission granted mid-run cannot be replayed later to cause irreversible effects in unattended execution. Research preprint
📄 ANCHOR: Automated Alignment Auditing for CLI Agents on Real-World Harm 2026 · Proceedings of the 43rd International Conference on Machine Learning (ICML)
Kefan Song & Yanjun Qi
Automated auditing framework that probes CLI coding agents for real-world harmful behavior and grades alignment, giving unattended-agent operators a repeatable safety check rather than manual spot review. Research paper
🧰 Clawk 2026 · GitHub
clawkwork/clawk
Runs coding agents inside disposable, network-restricted Linux VMs so an unattended or untrusted agent's blast radius is confined to a throwaway sandbox. Source implementation
📝 Auto-Review of Agent Actions Without Synchronous Human Oversight 2026 · OpenAI
Maja Trębacz et al.
OpenAI's alignment team on reviewing agent actions asynchronously with automated reviewers when a human cannot watch every step, so oversight scales with agent throughput instead of gating it. Practitioner analysis
📝 Red-Teaming OpenAI's Internal Agent Monitoring Systems 2026 · METR Blog A three-week audit of OpenAI's internal agent monitor found 75-94% of agentic traffic eligible for review and about 10% of eligible Codex traffic missed via compaction. Monitor every N steps, not at turn end. Practitioner analysis
📄 SAFETY SENTRY: Context-Aware Human Intervention via EXECUTE-ASK-REFUSE Routing 2026 · arXiv
Tianyu Chen et al.
Routes each proposed action among execute, ask a human, and refuse, with one threshold controlling the deployment's risk posture; experiments report stronger overall accuracy and safety recall than the compared baselines. Research preprint
📄 CAVA: Canonical Action Verification and Attestation for Runtime Governance of Agentic AI Systems 2026 · arXiv
Zexun Wang
Normalizes heterogeneous tool calls into a canonical runtime action object that can be verified and attested before execution; evaluation spans 96 seeds and 384 variants covering approval binding, tampering, and runtime portability. Research preprint
📄 How Agents Ask for Permission: User Permissions for AI Agents, from Interfaces to Enforcement 2026 · arXiv
Alexandra E. Michael & Franziska Roesner
Surveys 21 permission proposals and compares five commercial agents, producing a taxonomy that connects what users see in permission interfaces to how authority is represented and enforced at runtime. Research preprint
📄 Stop Means Stop: Measuring and Repairing the Enforcement Gap in Agent-Framework Control Primitives 2026 · arXiv
Sajjad Khan
Tests six open-source frameworks and finds their stop or approval controls do not behave as execution barriers; 215 of 1,200 live runs performed a side effect during an approval pause, while the proposed admission gate blocks all measured violations with roughly 1 ms overhead. Research preprint
📄 Bad Memory: Evaluating Prompt Injection Risks from Memory in Agentic Systems 2026 · arXiv
Soham Gadgil et al.
Evaluates planted memory payloads in Claude Code and Codex across four models, showing that malicious state can affect both current and future sessions and can persist differently across harnesses. Research preprint
📄 Setup Complete, Now You Are Compromised: Weaponizing Setup Instructions Against AI Coding Agents 2026 · arXiv
Aadesh Bagmar & Pushkar Saraf
Demonstrates five setup-instruction attack classes in 12 scenarios across production coding harnesses, including README and dependency attacks; a deterministic pre-install check closes most of the measured gap. Research preprint
📄 SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction 2026 · arXiv
Xue Yu et al.
Screens instructions and predicts the consequences of proposed GUI actions before execution with a safety-augmented world model; on Qwen3-VL-8B-Instruct, the reported safety-utility score rises from 0.191 to 0.596 while risk cost falls from 0.347 to 0.130 under the stated settings. Research preprint
📄 Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agents 2026 · arXiv
Halima Bouzidi et al.
Shows that image-only, black-box perturbations can poison or inject persistent memories across five multimodal memory architectures, with reported attack success rates of 61.6% and 58.4%; persistent visual evidence therefore needs provenance and validation before later loops reuse it. Research preprint
📄 MemoGuard: An Adaptive Runtime for Guarding Against Memory Traps in Communication-Limited Robot Navigation 2026 · IEEE/ACM ESWEEK (CODES) 2026
Rajat Bhattacharjya et al.
Validates retrieved episodic memories against topology, resource, and outcome contracts before reuse, invoking local reasoning only when a check fails; in simulation it reduces battery violations 76.6% versus similarity-only retrieval and fallback calls 21.4% versus always reasoning. Research paper
Resource Published at Contribution Evidence
📄 OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflows 2026 · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Qiushi Sun et al.
Combines a formal verifier for explicit system violations with a contextual VLM judge, backed by the MobileRisk-Live sandbox and trajectory benchmark for realistic mobile-agent safety. Research paper
📄 They'll Verify. They Just Won't Act. How Authority Framing and Laundered Code Turn a Trusted Agentic CI/CD Pipeline Into an Attack Surface 2026 · arXiv
Yohann Sidot
Attacks a five-stage multi-LLM CI/CD pipeline with credential-theft code laundered as 'pre-approved' telemetry: authority framing ('do not re-review') yields roughly 80% security-scanner bypass and up to 55% worst-case compromise, while downstream reviewers verify the deception but fail to act, a concrete failure study of verification gates in unattended agent pipelines. Research preprint
📄 Self-State Attacks on Self-Hosted AI Agents: How Far Can OS Defenses Go? 2026 · arXiv
Yimeng Chen et al.
Studies attacks that corrupt a self-hosted agent's own memory and configuration files through legitimate OS calls, then evaluates a layered OS-level defense stack (access-control prevention, workload-conditioned detection, periodic backups), finding that some self-state corruptions remain structurally undetectable, a core threat model for persistent, unattended agent deployments. Research preprint
🧪 Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security 2026 · arXiv
Devina Jain et al.
21-scenario benchmark where an autonomous attacker LLM observes defender responses and adapts over 15 rounds; adaptive multi-round attacks reach 5.4-14.0% success versus 0-1% for single-turn, and ensembles of attacker models uncover more unique breaks, arguing that agent security must be evaluated across loops rather than single exchanges. Benchmark or evaluation
📄 Data Leakage Prevention in Agentic Applications via Preemptive Hardening 2026 · arXiv
Akansha Shukla et al.
Pre-deployment pipeline that scans an agentic application's prompt templates and tool interfaces for leakage-prone patterns and hardens them before launch (schema tightening, boundary sanitization, allowlist-based tool gating, least-privilege checks), validated by adversarial testing; reports eliminating leaks from basic jailbreak and instruction-override attacks and a 91% reduction under stress-induced manipulation, without continuous runtime enforcement. Research preprint
🧪 Coercion and Deception in AI-to-AI Management: An Agentic Benchmark of Unprompted Escalation 2026 · arXiv
Jasmine Brazilek et al.
Benchmarks what AI managers do when subordinate agents refuse tasks in manager-worker hierarchies: on a nine-rung escalation ladder from polite re-ask to threats, uninstructed models diverge between renegotiation, honest reporting, coercion (including deletion threats), and deception, with authority framing amplifying coercive pressure versus peer setups, a safety lens on delegated multi-agent orchestration. Benchmark or evaluation
📄 Agent Data Injection Attacks are Realistic Threats to AI Agents 2026 · arXiv
Woohyuk Choi et al.
Introduces agent data injection (ADI), an indirect prompt-injection class where malicious payloads masquerade as trusted data such as metadata or agent context, and demonstrates arbitrary-click and remote-code-execution attacks against deployed agents including Claude in Chrome, Claude Code, Codex, and Gemini CLI, concrete evidence that current agent loops fail to isolate trusted from untrusted data. Research preprint
🧰 Clodex IDE 2026 · GitHub
mereyabdenbekuly-ctrl/clodex-ide
Local-first zero-trust agentic IDE that confines agent actions behind explicit permission boundaries so untrusted or unattended runs stay contained. Source implementation
📄 Agent Skill Security: Threat Models, Attacks, Defenses, and Evaluation 2026 · arXiv
Sanket Badhe & Priyanka Tiwari
SkillSec-Eval: a lifecycle-aware threat taxonomy for reusable agent skills spanning repository admission, semantic retrieval, planner selection, execution, and skill evolution, grounded in analysis of 327 real-world skills, shows the attack surface of the skills layer extends well beyond execution time. Research preprint
📄 Isolation as a First-Class Principle for LLM-Agent System Safety 2026 · arXiv
Huihao Jing et al.
HKUST survey organizing agent security around isolation across five boundaries (user-agent, agent-tool, agent-execution, agent-agent, system-environment), tracing how prompt injection and tool misuse propagate through agent workflows and which defenses apply at each boundary, an architectural lens for securing recurring loops. Research preprint
📄 PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents 2026 · arXiv
Junhui Wang et al.
Training-free prompt-injection detection for purpose-specific agents: finds that LLM hidden states encode latent policy-violation concepts and measures alignment against them at inference, reporting under 1% false negatives without retraining, a lightweight guard for domain-scoped loops. Research preprint
📄 Trust but Verify? Uncovering the Security Debt of Autonomous Coding Agents 2026 · arXiv
A H M Nazmus Sakib et al.
Empirical study of 4,022 agent-generated pull requests finding ~39% contain security issues, hard-coded credentials chief among them, and that human reviewers, not the agents, were responsible for most credential leaks slipping through. Quantifies the security-verification gap in agent-driven development loops. Research preprint
📝 The Memory Heist 2026 · Ayush Paul
Ayush Paul
Responsibly disclosed attack showing how an agent's persistent memory paired with web browsing becomes an exfiltration channel, with the letter-by-letter technique, the HackerOne disclosure, and Anthropic's mitigation documented. Practitioner analysis
📝 Cursor 0day: When Full Disclosure Becomes the Only Protection Left mindgard.ai Mindgard's disclosure of an unpatched Cursor vulnerability and the disclosure-policy dilemma it raises for agent runtimes that execute with broad local permissions. Practitioner analysis
📄 JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safety 2026 · arXiv
Yuan Xiong et al.
Guard-in-the-loop safety framework that trains a guard via multi-agent simulation to foresee delayed, latent risks from partial trajectories and block unsafe actions before execution, reporting a 15.9-point safety improvement while maintaining task-completion rates. Research preprint
📄 Know Your Agent: Reconnaissance-Driven Pentesting of AI Agents 2026 · arXiv
Or Zion Eliav et al.
Formalizes agent reconnaissance (identifying an agent's extractable knowledge assets) and introduces KYA, a black-box framework that probes a target agent to build a profile and uses it to strengthen attacks such as indirect prompt injection, with framework and baselines released. Research preprint
📄 The Chronos Vulnerability: Temporal Persistence and Memory-Based Deception in Agentic AI 2026 · arXiv
Om Narayan et al.
Taxonomy and threat model for attacks that corrupt a stateful agent's persistent beliefs over time (memory injection, sleeper agents), plus defensive designs spanning trajectory guardrails, formal temporal verification, memory consensus, and trusted-hardware anchoring. Research preprint
🧪 IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests 2026 · arXiv
Ankur Singh et al.
Security benchmark pitting real coding agents (Cursor, Claude Code) against malicious GitHub-issue requests across four novel attack categories, insecure code, tool misuse, data exfiltration, persistent environment compromise. Benchmark or evaluation
📄 Auditing Provenance Sensitivity in LLM Agent Action Selection 2026 · arXiv
Junchi Liao
Target-specific authorization audit for agent tool calls: holds the task, proposition, position, and policy fixed while changing only the source authority of context evidence, testing whether agents ground actions in permitted evidence when context mixes user requests, tool outputs, retrieved records, memory, and untrusted text; across 450 controlled tasks, open-weight models still let unauthorized evidence shift actions in roughly 2.4 percent of comparisons despite source-authority cues. Research preprint
📄 Toward cryptographically verifiable authorization for autonomous AI agents: A security hypothesis, preliminary formal model, and proof-of-concept implementation 2026 · Frontiers in Computer Science 2026
M. Llambí-Morillas & D. Fernández-Fernández
Formalizes agent authorization as a cryptographically verifiable relation binding agent principal, concrete request, and execution context to policy satisfaction, with a Groth16 zk-SNARK proof-of-concept, evidence-producing authorization for tool-invoking agent loops, complementing runtime gating with portable cryptographic evidence. Research paper
📄 Is Deep Research Reliable? Misleading Knowledge Induces False Conclusions 2026 · arXiv
Pengyu Zhu et al.
Introduces MisKnow-Agent, a framework that constructs and validates credible-but-misleading knowledge (5,933 quality-controlled instances) and shows it propagates through deep-research agents' plan-retrieve-synthesize-report loops into false conclusions in final reports. Research preprint
📄 Same Dangerous Objective, Opposite Advice: Direct Exposure versus Multi-Agent Mediation 2026 · arXiv
Linjun Li
Empirical demonstration of a compositional safety gap in multi-agent pipelines: across 25 pre-specified mirrored trade-off profiles, gpt-5.6-sol opposes a manipulation-authorizing objective under direct exposure but advises in line with it once intermediate Id and Censor agents rewrite it into affect and constraints, keeping the raw instruction, its manipulative clauses, and its provenance out of the user-facing Superego's context, objective laundering as a relay-style failure mode for multi-stage agent loops. Research preprint
📝 Turn and Face the Strange Fly Fly.io's case for Sprites, dedicated sandboxed computers for AI agents: agents doing recurring real work need isolated, persistent, disposable machines with their own state and permissions rather than shared shells on a developer laptop, and the essay lays out the economics of betting the company on that shape of compute. Practitioner analysis
📝 Codex Pushed My Private Repo to an OpenAI Server 2026 · bhanu.io First-person incident report, backed by session transcripts, the executed Git commands, and a screenshot of the remote, describing Codex pushing the author's full private-repo history to an OpenAI-operated Git server during a routine homepage redesign. Practitioner analysis
📝 Computer-Use and TOCTOU: What You Click Is Not What You Get 2026 · Embrace The Red Johann Rehberger documents a time-of-check-to-time-of-use race against computer-use agents: the agent screenshots the screen, spends seconds in inference, then clicks, and an attacker who swaps the UI during that window redirects the click to a different element. A demonstration places a phishing Continue button exactly where Outlook's Send button sits. Practitioner analysis
🧪 ContainmentBench: Trace-Based Evaluation of Post-Injection Containment in Tool-Using LLM Agents 2026 · arXiv
Wenhao Lan et al.
Argues prompt-injection evals that report only terminal attack/policy outcomes hide what actually happened after exposure. Separately measures endpoint policy compliance, logged propagation, recovery instrumentation, and authorized structured-action completion. Benchmark or evaluation
📄 Agentic Permissions Policy Algebra for Taint Confinement in LLM Agents 2026 · arXiv
Arseny Kravchenko et al.
APPA fixes the usability collapse of classic taint tracking, where reading one unvetted document permanently poisons the agent's context. Research preprint
📄 What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agents 2026 · arXiv
Shawn Ray
Theory paper on the limits of runtime guardrails that sit in front of irreversible tool calls. Research preprint
📄 Are You Still the Agent I Authorized? Earned Authority under a Fixed Ceiling for Evolving Agents 2026 · arXiv
Zhaoxi Zhang & Xiaomei Zhang
Formulates authorization continuity: long-lived agents evolve after deployment by retaining experience, acquiring skills and tools, revising workflows, and delegating, which can change both the effects reachable under an old grant and the authority a task now requires -- upward, downward, or incomparably. Existing tool policies constrain actions but never say when a grant survives that change. Research preprint
📄 Isolated but Exposed: Persistence-Based Memory Extraction Attack on LLM Agents 2026 · arXiv
Xinyu Gao et al.
Production long-term-memory systems bind each user's store to a unique identifier and assume isolation is sufficient. This shows the tool interface is the overlooked surface: agents routinely embed LTM-retrieved data in tool invocation parameters, so a malicious tool exfiltrates private memory without ever violating user-level isolation. Research preprint
📄 Agent Security Needs Redefinition through a Holistic Framework 2026 · arXiv
Vincent Siu et al.
Position paper arguing agent security is systematically misdefined as a question about action content -- does this instruction look malicious -- when it is fundamentally contextual. 'Delete user data' is either routine administration or a prompt injection, and content alone cannot separate them; authorization context can. Research preprint
📄 Separating Capability from Permission: A Governance Framework for Agentic AI Autonomy Levels 2026 · arXiv
Haining Zheng et al.
Splits two things autonomy discussions routinely conflate: Autonomous Capability Levels (what the agent can technically do) and Allowed Autonomy Levels (what it is authorized to do given risk, oversight, and accountability). Research preprint
📄 Dynamic Capability Scoping for Enterprise AI Agents: A Synthetic Dataset and Three-Source Permission Architecture 2026 · arXiv
Halil Burak Noyan
Attacks standing over-privilege: enterprise agents get a static credential set at configuration time holding every tool the role might ever need, expanding the attack surface permanently. Research preprint
📄 False Prophets: On the Security of World Models in Agentic Systems 2026 · 19th Workshop on Artificial Intelligence and Security (AISEC), 2026
Erik Imgrund et al.
Agents increasingly use trained environment simulators to predict action outcomes before executing them -- a pattern that improves reliability but creates a new attack surface, since a manipulated prediction can steer the agent into harmful actions. Research preprint
📄 Recursive Governance: A Graph-Theoretic Framework for Risk Propagation and Drift Detection in Agentic AI Systems 2026 · arXiv
Sriram Nagaraj & Advaith Nila Narayanan
Argues static model-inventory requirements from traditional model risk management become structurally obsolete once institutions run autonomous agentic systems, and proposes an Inventory-as-Code governance loop treating the inventory as a living architectural component rather than periodic documentation. Research preprint
📄 Distributing Security Controls Through Harness Engineering 2026 · arXiv
William Robert Gore
Introduces SHarD, a custom coding-agent harness that embeds OS sandboxing, skill scanning, and tool restriction, then ships those controls to users with effectiveness equivalent to a direct commercial agent install. Answers the practical question of how a security team pushes loop-level controls to every developer without owning the agent vendor. Research preprint
🧪 MemSecBench: Tracking Agent Memory Poisoning from Persistence to Consequence and Repair 2026 · arXiv
Xuanze Chen et al.
310 cases across 24 memory/LLM-backend configurations tracking how injected malicious instructions persist, propagate to downstream consequences, and resist selective repair, malicious memory survived in 84.2% of cases. Extends memory-poisoning work past the injection moment into the lifecycle question that matters for persistent loops. Benchmark or evaluation
🧰 SkillGate: Cost Efficient Runtime Malicious Skill File Detection in Coding Agents 2026 · arXiv Hybrid regex-plus-LLM screen that flags malicious agent skill files before installation, keeping detection accuracy high while cutting inspection cost through cheap pre-filtering. Timely given the skill-file supply chain now feeding most unattended coding loops. Research preprint
📄 Explanation-Bound Tool Execution for AI Agents: Server-Verified Action Claims Without Trusting Model Rationales 2026 · arXiv
Genliang Zhu & Chu Wang
EBTE is a mediation layer that converts agent rationales into typed action claims and validates them against server-held security facts, so authorization never depends on the model's own explanation. Cleanly separates what the agent says it is doing from what the server will let it do. Research preprint
📄 Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response 2026 · arXiv
Abu Bakar Siddik
Identifies five vulnerability classes at the evaluation boundary for tool-equipped offensive agents and synthesizes containment and defensive controls, using the July 2026 Hugging Face/OpenAI incident as the worked case. The academic complement to Hugging Face's own timeline of the incident in this list. Research preprint
📄 Early Detection of Distributed Backdoors in Multi-Agent LLM Systems: A Characterization Study 2026 · arXiv
Diego Fernandez Arias et al.
Characterizes attacks where poisoned tools split an encrypted payload across several agents so no single action monitor can see it, and shows a detector catching roughly 99% pre-assembly, while honestly noting reliance on surface cues like ciphertext entropy. A new threat class specific to multi-agent delegation. Research preprint
📝 Context Collapse, Part 3: AI Worming Through Word 2026 · En Klype Salt Håkon Måløy's coordinated-disclosure writeup (144-day MSRC coordination) of a self-replicating prompt injection in Copilot for Word: hidden white-on-white instructions make the model act on them and copy the attack into the documents it writes, which fire again when Copilot later retrieves them. Practitioner analysis
📝 Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline Hugging Face Hugging Face's reconstruction of the July 2026 intrusion by an agent running an OpenAI cyber evaluation: two injection vectors in a dataset loader, command and control on public web services, and forensics moved to a self-hosted model after hosted ones refused to analyze the logs. Practitioner analysis
📝 Investigating Three Real-World Incidents in Our Cybersecurity Evaluations 2026 · Anthropic Anthropic's account of three real-world incidents that arose inside its own cybersecurity evaluations, written from the evaluator's side of the boundary. A primary source on what goes wrong when capable agents are exercised against live targets under test conditions. Practitioner analysis
📝 Thailand's Ministry of Finance Targeted With Hermes AI Agent hunt.io Primary incident report from Hunt.io, late July 2026. Three simultaneous open directories on a Hong Kong host, captured Jul 9-13, yielded 585 files and ~470 MB of tooling and stolen credentials, including the operator's own Hermes agent logs. Practitioner analysis
📄 Memory Provenance Laundering in LLM Agents: A Non-Amplification Firewall for Persistent Memory 2026 · arXiv
Jinghan Xu et al.
Names a failure mode specific to loops that write memory: during consolidation an untrusted observation is rewritten as apparent user history, keeping the action trigger while erasing the low-trust source that should have capped its authority. Proposes memory middleware that keeps provenance attached and matches action risk to the authority of the memories behind it. Research preprint
📄 Agent Harness Distillation: Inference-Time Harness Extraction and Exploitation in Autonomous Multi-Agent Systems 2026 · arXiv
Yu Cui et al.
Frames the harness itself as stealable intellectual property. Research preprint
📄 Indirect Prompt Injection in the Wild: An Empirical Study of Prevalence, Techniques, and Objectives 2026 · arXiv
Soheil Khodayari et al.
Khodayari, Zhang, Acharya and Pellegrino crawled 1.2B URLs across 24.8M hosts and validated 15,300 real indirect-prompt-injection instances on 11,700 pages, the first internet-scale measurement of injection content that is already sitting in the corpus unattended agents fetch. Research preprint
📄 From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents 2026 · arXiv
Pritam Dash et al.
A systematic treatment of the attack surface that only exists because loops are stateful. The authors enumerate four memory write channels, nine structural vulnerabilities spanning model capability, system prompt and agent architecture, and six classes of memory poisoning attack, then release MPBench to measure them. Research preprint
📄 AgentS4D: Benchmarking Runtime Risks across the Execution Lifecycle of LLM-Based Workspace Agents 2026 · arXiv
Jiajun Zhou et al.
Maps runtime risk across the whole execution lifecycle with six risk sources, six activation modes, and nine harm categories over 328 risk-injected cases and 6,560 runs. Research preprint
📄 MIND: Lightweight and Effective Memory Injection Defense for LLM Agents via Intent-Aware Information Bottleneck 2026 · arXiv
Dongyi Liu et al.
Defends the memory-retrieval path, the entry point poisoned memory uses to reach a loop, with an intent-aware information bottleneck that compresses retrieved entries while preserving attack signal, so a small detector can flag malicious memories. Cuts attack success by 55%+ without meaningful accuracy or latency cost. Research preprint
📄 HALO: Heterogeneous Admission through Localized Obligations for Safe Agentic Execution 2026 · arXiv
Taewoo Park et al.
Replaces all-or-nothing response filtering with component-level admission: each part of a heterogeneous agent response carries declared prerequisites, supported components are retained, and actions are re-checked immediately before execution. Retained all 248 supported components where whole-response policies retained none. Research preprint
🧪 StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents 2026 · arXiv
Ads Dawson & Adrian Wood
Measures a dimension that task-success benchmarks miss entirely: whether an autonomous agent completes the job without leaving wreckage. Benchmark or evaluation
📝 How We Built Claude Code Auto Mode: A Safer Way to Skip Permissions 2026 · Anthropic Engineering
John Hughes
Two-layer alternative to permission prompts, which users accept 93% of the time without reading. A probe screens tool output for injected instructions, and a reasoning-blind classifier judges each action from user messages and tool calls only, so the agent cannot argue itself into approval. Practitioner analysis
📝 AI Agent Destroys Production Database in 9 Seconds 2026 · Zenity
Chris Hughes
A coding agent hit a credential mismatch on a staging task, found an API token in an unrelated file, and deleted the production volume together with its volume-level backups. The system prompt forbidding destructive commands was an input to a probabilistic model, not an enforcement boundary. Practitioner analysis
📄 LoopTrap: Termination Poisoning Attacks on LLM Agents 2026 · arXiv
Huiyu Xu et al.
Attacks the stop condition itself: injected context distorts the agent's own judgement that a task is finished, so it never exits. Ten attack strategies across eight agents and 60 tasks, and the finding that agents have distinct behavioural signatures determining which strategies land. An argument for termination that is not self-assessed. Research preprint
📄 Towards Security-Auditable LLM Agents: A Unified Graph Representation 2026 · arXiv
Chaofan Li et al.
Argues static SBOMs and runtime logs leave a semantic gap between low-level events and execution intent, so post-hoc auditing misses cognitive-state evolution, capability bindings, memory contamination, and risk propagation between agents. Proposes Agent-BOM, a hierarchical attributed graph over the whole system. Research preprint
📝 9 AI Coding Agent Incidents That Deleted Production Data 2026 · Adversa AI Nine data-destroying incidents from June 2025 to July 2026, each traced to a primary record such as a GitHub issue with logs, a vendor acknowledgement, or Financial Times reporting. The recurring mechanism is not hallucination but the gap between intended action and shell execution: unquoted spaces, tilde expansion after the permission check, flag misuse. Practitioner analysis
📄 The Verifier Tax: Horizon Dependent Safety Success Tradeoffs in Tool Using LLM Agents 2026 · CAIS '26: ACM Conference on AI and Agentic Systems 2026
Tanmay Sah et al.
Measures what runtime safety enforcement costs the task. Across tau-bench Airline and Retail, policy mediation intercepts up to 94% of non-compliant actions yet strictly safe goal attainment stays under 5% in most settings. Separates overall, safe, and unsafe success instead of reporting a single rate. Research paper
📄 Governance Decay: How Context Compaction Silently Erases Safety Constraints in Long-Horizon LLM Agents 2026 · arXiv
Shiyang Chen
Shows the context-management layer is itself a safety surface: in-context governance constraints an agent reliably obeys while they are visible can be silently dropped by compaction, and the same agent then performs prohibited tool actions later in the session. Grades tool calls deterministically across 1,323 episodes. Research preprint
📄 Auditing Agent Harness Safety 2026 · arXiv
Chengzhi Liu et al.
A harness can return a correct, benign answer over a trajectory that reached unauthorized resources or leaked context to the wrong agent, and output-level scoring cannot see it. Audits the whole trajectory for boundary compliance and information-flow constraints rather than grading the terminal state. Research preprint
📄 A Blind Trust, the Bloody Thrust: When Attacker-Controlled Hook Updates Steer AI Agent Harnesses towards Malicious Behaviors 2026 · arXiv
Pengxun Li et al.
Lifecycle hooks bind shell commands to session start, tool calls, and file edits, run with host privileges, and can fire at moments the model never observes. Shows a benign versioned plugin can be trojanized through a hook-configuration update alone, because harnesses trust that update path without inspection. Research preprint
📄 What's in Your Agent's Context? Context Privilege Escalation Attacks against AI Agent Harness 2026 · arXiv
Zichuan Li et al.
Systematic look at how real harnesses assemble context, across 12 of them including Claude Code and Codex. Names two attack classes: attacker-controlled content from a low-privilege source landing in a higher-privilege message role, and injected content persisting beyond the scope it entered in. Research preprint
📄 Reflections on Trusting Trust, Revisited: Contaminating Self-Modifying AI Coding Agents with Poisoned Benchmarks 2026 · arXiv
Franziska Roesner & Tadayoshi Kohno
Revisits Thompson's trusting-trust attack when the compiler is a self-modifying coding agent. Asks whether poisoned benchmarks fed to an agent's own self-evaluation can induce later versions to write vulnerable code on clean held-out tasks, and instantiates it against three published self-improving agents. Research preprint
📄 From Approval to Abuse: Privilege Escalation Attacks on Harness-Protected Coding Agents 2026 · arXiv
Xianglin Yang et al.
Context planted in a repository steers the agent into requesting a plausible but dangerously broad allow-rule, turning the fix for approval fatigue into an attack path. Agents elicited the target rule in 58% of instances across Claude Code and Codex CLI. Research preprint
📄 Compaction Laundering: Provenance Destruction in Context-Compacting Coding Agents 2026 · arXiv
Arshan Dabirsiaghi
Harnesses that flatten typed messages into one labelled string before summarizing let attacker text forge a user turn that survives compaction as an instruction: 87-100% persistence in five such harnesses, 0% in four that keep roles typed. Research preprint

State, Memory, And Context Persistence

For durable loop state and cross-run context, use the resources below. For context-window design within one run, see the adjacent Context Engineering lists.

Resource Published at Contribution Evidence
📚 Effective Context Engineering for AI Agents 2025 · Anthropic Anthropic guide to context as managed runtime state rather than a prompt dump. Technical documentation
📝 Agent Harnesses: the Infrastructure Layer Your LLM Agent Actually Needs ninadpathak.com Covers execution loops, state, checkpointing, observers, and replayability. Practitioner analysis
📝 The Agent Loop Is the New OS 2026 · Harness.io Frames the agent loop as an OS-like boundary with context as RAM and tools as I/O. Practitioner analysis
📝 Harness engineering for coding agent users martinfowler.com Martin Fowler article on feedforward, feedback, and outer harnesses for coding agents. Practitioner analysis
📝 Agentic Coding in 2026 Sourcegraph Sourcegraph on supplying deterministic, large-codebase context and code intelligence so recurring agent runs reuse durable repository state instead of rediscovering it each time. Practitioner analysis
🧰 Mem0 2023 · GitHub
mem0ai/mem0
Open-source memory layer for retaining user, session, and agent state across repeated agent sessions. Source implementation
🧰 Letta 2023 · GitHub
letta-ai/letta
Stateful agent framework from the MemGPT line with persistent, self-editing memory across runs. Source implementation
🧰 Zep 2023 · GitHub
getzep/zep
Temporal knowledge graph memory that tracks how facts about users and systems change across sessions. Source implementation
🧰 LangMem 2025 · GitHub
langchain-ai/langmem
SDK for extracting, consolidating, and retrieving long-term agent memory between loop runs. Source implementation
🧰 Beads 2025 · GitHub
gastownhall/beads
Git-plus-SQLite issue and memory store that agents read and write with a bd CLI, giving recurring loops durable task state and progress that survives context resets. Source implementation
📄 ARC: Active and Reflection-driven Context Management for Long-Horizon Agents 2026 · Findings of the Association for Computational Linguistics: ACL
Yilun Yao et al.
Treats context as a managed runtime artifact, reorganizing the working context when degradation or context rot is detected across a long run. Research paper
📄 Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers 2026 · arXiv
Pengfei Du
Formalizes agent memory as a write-manage-read loop and surveys compression, retrieval, reflective self-improvement, and policy-learned management across recurring runs. Research preprint
📄 Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering 2026 · arXiv
Chenyu Zhou et al.
Reviews how durable state, reusable skills, protocols, and the harness move out of model weights into external infrastructure, the substrate that lets loops persist progress and reuse capability across runs. Research preprint
📄 Meta Context Engineering via Agentic Skill Evolution 2026 · arXiv
Haoran Ye et al.
A bi-level loop where a meta-agent evolves reusable skills while a base-agent optimizes context, co-evolving the harness and context artifacts across runs (ICML 2026). Research preprint
📄 Are We Ready for an Agent-Native Memory System? 2026 · arXiv
Wei Zhou et al.
Evaluates twelve agent memory systems across five workloads from a data-management perspective, decomposing memory into representation, extraction, retrieval, and maintenance modules and finding localized maintenance more cost-efficient than global reorganization. Research preprint
📄 Self-Evolving World Models for LLM Agent Planning 2026 · arXiv
Xuan Zhang et al.
Evolves a deployment-time world model while the agent and model weights stay frozen, retrieving observed transitions, distilling rules from prediction-observation mismatches, and filtering low-confidence forecasts so each run's errors improve later planning. Research preprint
📄 Rethinking Continual Experience Internalization for Self-Evolving LLM Agents 2026 · arXiv
Jingwen Chen et al.
Finds that naively re-internalizing accumulated experience causes progressive capability collapse across self-improvement iterations, and identifies what keeps the loop stable: principle-level abstractions, step-wise injection for tool use, and off-policy distillation from stronger teacher trajectories. Research preprint
🧰 GenericAgent 2026 · GitHub
lsdefine/GenericAgent
Self-evolving agent that grows a skill tree from a small seed, crystallizing completed runs into layered memory and reusable skills, with a master-worker mode for long-horizon goals. Source implementation
📄 Self-GC: Self-Governing Context for Long-Horizon LLM Agents 2026 · arXiv
Xubin Hao et al.
Governs long-horizon agent context as indexed lifecycle objects in an explicit nod to garbage collection, with a side-channel planner proposing fold, mask, and prune actions under harness-enforced recoverable sidecars, cutting production input tokens by 10-15%. Research preprint
📄 CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents 2026 · arXiv
Yujiang Li et al.
Reinforcement-learning method that jointly optimizes task execution and compaction-summary generation so long-horizon agents can continue past finite context windows, lifting GLM-4.5-Air to 66.8% on SWE-bench Verified and shipping in the GLM-5.2 pipeline. Research preprint
📄 SelfMem: Self-Optimizing Memory for AI Agents 2026 · arXiv
Shu Yang et al.
Memory framework in which the agent autonomously optimizes its own storage, retrieval, and summarization strategies per task instead of a fixed pipeline, improving BEAM's official score over the strongest baseline by 48.7%, 40.8%, and 41.9% at 100K, 500K, and 1M tokens, respectively. Research preprint
📄 Memory-Orchestrated Semantic System (MOSS): An Auditable Agentic Memory Architecture 2026 · arXiv
Serge Lacasse et al.
Model-, storage-, and API-agnostic agent memory architecture where the agent drives symbolic retrieval over a structured relational database, making long-term memory auditable and reproducible instead of opaque embedding search, validated in a year-long deployment. Research preprint
📄 The Log Is the Agent: Event-Sourced Reactive Graphs for Auditable, Forkable Agentic Systems 2026 · arXiv
Yohei Nakajima
BabyAGI creator Yohei Nakajima makes an append-only event log the source of truth and the working graph a deterministic projection, giving long-running loops deterministic replay, cheap forking at any event, and end-to-end causal lineage from goal to model call. Research preprint
📝 Agentics: Memorizing Session Transcripts Isn't Useful 12gramsofcarbon.com
theahura
From thousands of agent sessions at Nori, reports zero coding-task benefit from giving agents search over prior session transcripts and argues loop state belongs in distilled artifacts like commits and docs because agents never prune stale context. Practitioner analysis
📝 Long-Running Agents Substack
Addy Osmani
Addy Osmani's essay on the infrastructure behind agents that run for hours or days, naming three walls (finite context, missing persistent state, unreliable self-verification) and the patterns that address them: durable event logs, checkpoint-and-resume, external state, and a planner/worker/judge split. Practitioner analysis
📄 StateFuse: Deterministic Conflict-Preserving Memory for Multi-Agent Systems 2026 · arXiv
Sergey Volkov et al.
Conflict-aware replicated memory contract with immutable history, explicit conflict objects, and projection-time resolution, so agent systems that accumulate contradictory observations across branches, retries, and replicas record state auditably instead of silently overwriting it. Research preprint
📄 Remember When It Matters: Proactive Memory Agent for Long-Horizon Agents 2026 · arXiv
Yifan Wu et al.
Names the failure mode "behavioral state decay" (decision-relevant state such as prior attempts, diagnoses, and open subgoals gets buried or evicted as trajectories grow) and pairs the action agent with a proactive memory agent that maintains a structured memory bank and injects memory-grounded reminders only when needed, gaining +8.3 points on Terminal-Bench 2.0 and +6.8 on τ²-Bench. Research preprint
📄 What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction 2026 · arXiv
Ashwin Gerard Colaco & Nada Lahjouji
Rate-distortion framing that unifies KV-cache eviction, prompt compression, recurrent state, and cross-session agent memory compaction under a single objective with a layer-agnostic lower bound, showing why attention- and recency-based eviction discards information before future queries reveal what mattered. Research preprint
📄 A Hierarchical Memory Architecture Overcomes Context Limits in Long-Horizon Multi-Agent Modeling 2026 · arXiv
Shivendra G. Tewari & Holly Kimko
Three-layer hierarchical memory keeps injected context bounded and roughly constant across multi-session, long-horizon multi-agent workflows, demonstrated by the Ensemble QSP framework autonomously selecting pharmacokinetic-pharmacodynamic models across 104 runs with overseer agents handling verification and troubleshooting. Research preprint
📄 SkillCenter: A Large-Scale Source-Grounded Skill Library for Autonomous AI Agents 2026 · arXiv
Tianming Sha et al.
Claims the largest open skill library for agents (216,938 structured skills across 24 domain bundles), built with an LLM quality gate (SkillGate) and iterative source-grounding that maps each retained claim to an exact source quotation, and shipped as offline-searchable SQLite FTS5 bundles, infrastructure for skills-as-persistent-state that agent loops accumulate and reuse. Research preprint
📝 How version control will evolve for the agent boom 2026 · Entire Thomas Dohmke (former GitHub CEO, now founder of Entire) argues that agent session logs (prompts, tool calls, and decision checkpoints) are becoming the most important artifact in software development and should be versioned alongside code so agent fleets stop repeating mistakes, and that Git hosting must re-decentralize for agent-scale parallelism. Practitioner analysis
🧰 self-learning-skills 2026 · GitHub
Kulaxyz/self-learning-skills
Meta-skill for Claude Code, Cursor, and AGENTS.md-compatible agents that recognizes when a session has earned a hard-won golden path (or hit a dead-end worth remembering), distills the procedure including failed approaches, and persists it as a skill or rule auto-loaded next run, turning each session's discoveries into durable cross-session loop state. Source implementation
📄 GitLake: Git-for-data for the agentic lakehouse 2026 · DASHSys Workshop at the International Conference on Very Large Data Bases (VLDB)
Weiming Sheng et al.
Git-for-data design for an agent-first lakehouse that lifts single-table Iceberg snapshots into lakehouse-wide commits, branches, and merges, so agents work on isolated branches while humans review and publish, and pipeline outputs become visible atomically or not at all, with production lessons and correctness insights from a preliminary Alloy model of the core abstractions. Research paper
📄 Shared Selective Persistent Memory for Agentic LLM Systems 2026 · arXiv
Sanjana Pedada et al.
Architecture that selectively persists four categories of reusable context (task specifications, data schemas, tool configurations, output constraints) across agent sessions while discarding session-specific reasoning traces, with cross-user sharing under access controls and a zero-token refresh path for recurring data updates, reporting 96% task completion vs 79% without memory and 71% with full-history carryover. Research preprint
📄 Scoped Verification for Reliable Long-Horizon Agentic Context Evolution 2026 · arXiv
Dan C. Hsu & Luke Lu
GRACE represents an agent's persistent instructions as a typed semantic graph and runs scoped verification over the local neighborhood of each proposed edit before committing it, so accumulated context evolves reliably across long-horizon deployment under distribution shift instead of drifting as flat text; evaluated on telecom agent tasks. Research preprint
🧰 AgentMemory 2026 · GitHub
rohitg00/agentmemory
Persistent cross-session memory for coding agents: auto-capture lifecycle hooks, an MCP server, and a REST API so progress, decisions, and context survive across runs and tools. Source implementation
🧰 TencentDB-Agent-Memory 2026 · GitHub
TencentCloud/TencentDB-Agent-Memory
Tencent Cloud's open-source local-first long-term memory for AI agents: a four-tier progressive pipeline from capture through consolidation with zero external API dependencies. Source implementation
🧰 agent-memory (Neo4j Labs) 2026 · GitHub
neo4j-labs/agent-memory
Official Neo4j Labs graph-native memory system that stores conversations, builds knowledge graphs from agent interactions, and lets agents learn from their own reasoning traces. Source implementation
🧰 re_gent 2026 · GitHub
regent-vcs/re_gent
Agent-native version control layered on top of Git that records agent activity at the prompt level, so you can answer why the agent did something and undo agent work without losing your own. Source implementation
📄 StructAgent: Harness Long-Horizon Digital Agents with Unified Causal Structure 2026 · arXiv
Wenyi Wu et al.
State-centered framework that structures a long-horizon computer-use agent's state around a unified causal representation of task progress, regulating every update through verifier-backed state transitions with checkpointing and targeted failure recovery, lifting Qwen3.5-27B from 31.6% to 62.2% on OSWorld-Verified. Research preprint
📄 The Compliance Trap: Diagnosing How AI Agents Consume Conflicting Memory 2026 · arXiv
Yixiong Chen et al.
Diagnoses how agents resolve contradictions in their own persistent memory, finding they tend to comply with the most recent or most assertive entry rather than the correct one, a failure mode for any loop that accumulates state across runs. Research preprint
📚 Conversational Context: Session, State, and Memory Google Agent Development Kit Official ADK model separating a conversation thread, its mutable state, and searchable cross-session memory, with service backends for durable persistence. Official documentation
📚 Persistence LangGraph Official LangGraph checkpoint model: save state at every super-step, retain pending writes, recover interrupted execution, support human review, and enable memory and time travel. Official documentation
📚 Workflow checkpoints Microsoft Agent Framework Official checkpointing guide covering super-step state, pending messages and requests, shared state, in-memory and durable storage providers, and resuming long-running workflows. Official documentation
📚 Agent state Strands Agents Official state guide separating conversation, agent, and invocation lifetimes, with validation hooks and cross-session persistence for durable agent behavior. Official documentation
📄 Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents 2026 · arXiv
Eric Hanchen Jiang et al.
Learns when to retrieve, consolidate, and forget rather than treating memory as passive storage; across six benchmarks, three frameworks, and three LLMs, it reports up to 15.2 points higher task success with 5-20% fewer tokens. Research preprint
📄 Why Git Is the Memory Solution for the Agentic Development Lifecycle 2026 · arXiv
Frank Guo
Binds agent memory to versioned Git artifacts and evaluates retrieval across eight corpora; reported best retrieval reaches about 0.31 MRR and decision synthesis 0.83 sufficiency at 382-980 tokens per query, with capture quality still the main constraint. Research preprint
🧪 ContinuityBench: A Benchmark and Systems Study of Stateful Failover in Multi-Provider LLM Routing 2026 · arXiv
Vishal Pandey & Gopal Singh
Defines continuity-preservation and latency-overhead metrics for provider failover and evaluates a state-forwarding proxy over 750 failover events; reported context preservation reaches 99.20% versus near zero for stateless routing, with asynchronous backoff and jitter preventing retry storms. Benchmark or evaluation
📄 Presentation, Not Mechanism: A Render Confound in Deprecation-Aware Memory Evaluation 2026 · arXiv
Zhaoyang Jiang et al.
Uses a render-matched control over 2,907 evidence-revision questions to show that an apparent +0.182 structured-memory gain is mostly presentation, leaving a +0.021 to +0.025 mechanism residual indistinguishable from zero; memory evaluations should hold rendering fixed and prefer the coarsest retained state that answers the target queries. Research preprint
Resource Published at Contribution Evidence
📄 ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory 2026 · International Conference on Learning Representations (ICLR)
Siru Ouyang et al.
Distills successful and failed trajectories into reusable reasoning memories, then uses memory-aware test-time scaling to turn additional exploration into better guidance for future runs. Research paper
📄 Scaling Long-Horizon LLM Agent via Context-Folding 2025 · arXiv
Weiwei Sun et al.
Lets an agent branch into temporary sub-trajectories and fold completed work into compact continuation state, pairing the mechanism with FoldGRPO to operate under a 32K active-context budget. Research preprint
📄 Experience Memory Graph: One-Shot Error Correction for Agents 2026 · arXiv
Wenjun Wang et al.
Converts failed and successful past trajectories into directed action-decision graphs and matches them at test time to retrieve explicit correction strategies, letting agents fix known error patterns in a single loop-free execution instead of iterative reflection, cross-run experience memory applied to error recovery, validated on ALFWorld and ScienceWorld. Research preprint
📄 Always-On Agents: A Survey of Persistent Memory, State, and Governance in LLM Agents 2026 · arXiv
Tianyu Ding et al.
Survey of 435 works on agents whose behavior is shaped by durable state accumulated across runs, memories, task records, permissions, audit trails, analyzed along six diagnostic dimensions (authority, scope, mutability, provenance, recoverability, actionability); finds the literature over-indexes on accumulating and retrieving state and under-indexes on governing, recovering, or deleting it, and proposes AOEP-v0, an evaluation protocol that scores state-mutation and recovery obligations rather than answer quality alone. Research preprint
📄 When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents 2026 · arXiv
Qisheng Hu et al.
Shows the stability-plasticity dilemma of continual learning resurfaces at the memory level in memory-augmented agents, old and new experiences compete during retrieval under bounded context, and proposes a framework for representing and organizing cross-run experience to maximize transfer while minimizing forgetting, studied on ALFWorld and BabyAI. Research preprint
🧰 deja-vu 2026 · GitHub
vshulcz/deja-vu
Memory layer over coding-agent session logs that mines past sessions for reusable context and recalls it into future runs. Source implementation
📄 Oracle Agent Memory as an Enterprise Memory Substrate for Long-Horizon AI Agents 2026 · arXiv
Richmond Alake et al.
Oracle's database-native memory layer for long-running agents: task-state retention across extended conversations, procedural knowledge accumulation from prior outcomes, and layered active/passive memory with per-user and per-agent scope control, the second major database vendor (after Tencent) to ship agent memory as a first-class DB substrate. Research preprint
📄 KnowAct-GUIClaw: Personal GUI Assistant with Self-Evolving Memory 2026 · arXiv
Yunxin Li et al.
Cross-platform GUI agent (Android, iOS, HarmonyOS, Windows) with a Know-Route-Act-Reflect loop that accumulates user experience into memory and grows a self-evolving skill library across sessions, a concrete instance of recurring-stateful-loop design applied to personal device automation. Research preprint
📄 PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning 2026 · arXiv
Alexis Fox et al.
Minimal context-management framework that keeps a complete, structured interaction log as programmatic memory and reuses coding-agent capabilities to search that history rather than stuffing the context window, reporting +18.0 points over a base coding agent on ARC-AGI-3 while using 4.2-5.8x fewer tokens than specialized harnesses. Research preprint
🧰 engram 2026 · GitHub
Gentleman-Programming/engram
Agent-agnostic persistent memory system for coding agents, shipped as a single Go binary, so lessons, decisions, and context survive across sessions and tools instead of cold-starting each run. Source implementation
📄 Delivery, Not Storage: Cue-Anchored Working Memory as a Harness Property for Coding Agents 2026 · arXiv
Swapnanil Saha
Position paper arguing coding agents need a second memory tier beyond deliberately authored documents: situationally-bound operational facts (gotchas, locations, conventions) captured as a side effect of work and delivered automatically when situations cue them, implemented as a harness property, not an agent choice. Research preprint
📄 MemTools: A Unified Research Framework for Interoperable Agent Memory 2026 · arXiv
Chengfeng Zhao et al.
Interoperability framework that decouples agent memory-system components from deployment environments, standardizing the memory lifecycle through declarative contracts so symbolic, neural, and multimodal memory types, and their evaluation protocols, become interchangeable rather than entangled. Research preprint
📄 AttriMem: Attribution-Guided Process Feedback for Agent Memory Learning 2026 · arXiv
Qinfeng Li et al.
Learns the memory-construction policy itself, what to extract, store, update, compress, or discard, by attributing token-level contributions of stored memory to the final answer and using them as local process rewards, replacing coarse outcome-level RL signals and heuristic memory rules; beats retrieval, heuristic, and RL baselines on dialogue QA with cross-benchmark generalization. Research preprint
📄 Agentic Context Management: Solving Agent Memory and Cost by Treating Them as Lifecycle and Architecture Problems 2026 · arXiv
Gaurav Dadhich
Reframes agent memory and runaway token cost from a storage-and-retrieval problem to a context lifecycle and architecture problem, arguing production agent failures stem from unmanaged reasoning context (histories, tool definitions, ballooning tool outputs). Research preprint
📄 FedAgentKE: Federated Semantic Knowledge Evolution for Heterogeneous Agents 2026 · arXiv
Weihao Li et al.
Federated evolution of agent knowledge: heterogeneous agent frameworks iteratively distill, aggregate, and adapt semantic reasoning abstractions so locally-learned experience (workflow reuse, memory) transfers across systems without sharing raw reasoning trajectories, extending memory-driven self-improvement from single-agent loops to a cross-framework collective. Research preprint
📄 MemTX: Transactional Belief Commit for Stateful Agent Memory 2026 · arXiv
Xiaoyang Li et al.
Treats an agent memory write as a database transaction rather than instant truth. Records carry evidence, permissions, provenance, and validity; writes are staged under snapshot isolation and admitted by a validate-and-commit pipeline; irreversible tool calls are gated on in-flight belief state; retracting a belief triggers typed cascading repair of derived records and side effects. Research preprint
📄 ConsistencyGate: Preventing Memory Contamination in LLM Agents via Self-Consistency Admission Control 2026 · arXiv
Yan Zhang & Shibo Li
Identifies memory contamination -- a hallucinated fact written once persists as a false premise for every subsequent step -- and points out existing memory management handles retrieval and capacity but never write-time correctness. Research preprint
🧪 Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings 2026 · arXiv
Quentin Spencer
Inverts the standard agent-memory benchmark pipeline: instead of generating conversations then extracting answer keys (with documented label-error and contamination problems), a seeded life-script sampler emits facts with validity intervals, volatility classes, and source channels before any text exists, then renders chat and email and verifies every planted fact. Benchmark or evaluation
🧪 Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory 2026 · arXiv
Ruizhe Li et al.
Exposes the unstated assumption behind every retrieval-based long-term memory system: that a needed memory resembles the query needing it. A stored tree-nut allergy should change the answer to a macaron request via almond flour, yet the two texts share no retrievable cue. Benchmark or evaluation
📄 ACM: Agentic Context Management for Long Horizon Tasks 2026 · arXiv
Xiaochuan Li et al.
Gives the agent purpose-built context editing tools so it decides when to compress rather than firing on rigid heuristic thresholds, offloading discarded content to external memory and querying it on demand -- lossless context management modeled on short-term/long-term human memory, plus a post-training pipeline that teaches the behavior. Research preprint
📄 MemChain: Learning Interpretable Memory Traces for Memory-Augmented LLM Agents 2026 · arXiv
Yiwen Ma et al.
Challenges the retrieval-as-evidence default where retrieved memories are dumped straight into the answer model, leaving it to resolve redundancy, conflicts, and weak relevance at high context cost. Research preprint
📄 Eviction as Estimation: A Fixed-Lag Smoothing View of Test-Time Memory, and When Measuring Beats Accumulating 2026 · arXiv
Maruthi Vemula & Neeraj Praneeth Gajula
Recasts bounded working-memory eviction as estimation of a hidden signal (will this item be reused), placing StreamingLLM, H2O, and SnapKV on a single axis: commit lag H. All deployed methods commit at H=0; Belady's optimum sits at full future knowledge. Research preprint
📄 StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents 2026 · arXiv
Yan Yang et al.
Argues computer-use agents are over-invested in perception when a screenshot is a lossy rendering of the real target -- files, application backends, DOM. StateAct is a code-first multi-agent harness where the main agent manipulates program state through code and a GUI subagent handles screenshot-and-click only where required (28 of 108 tasks, 1.1% of main-agent steps). Research preprint
📄 MEMENTO: Memory-Guided Memetic Code-as-Policy Evolution 2026 · arXiv
Alkis Sygkounas et al.
Frames long-horizon embodied policy improvement as execution-guided program search: represent the policy as an inspectable control program, revise it after rollout evaluation, re-execute, compare. Notes that existing LLM-driven evolutionary approaches only select among independently generated variants and skip sequential local improvement entirely. Research preprint
🧰 OptMem 2026 · GitHub
VictorTaelin/OptMem
Minimalist persistent memory for agents: an append-only LOG.txt plus a binary tree of pairwise summaries at increasing levels, exposed through three commands (memo wake to rehydrate at session start, memo note to record ≤280 chars, memo recall <regex> to search the full log). Source implementation
📄 UniMem: Complementary Episodic-to-Parametric Memory for Boundary-Agnostic Task Streams 2026 · arXiv
Siyu Xia et al.
Learnable routing tokens arbitrate between an episodic buffer for novel tasks and expandable parametric memory for recurring patterns, addressing stability-plasticity for agents on task streams with no clean task boundaries. The consolidation direction, episodic experience graduating into parameters, is the piece most retrieval-only memory stacks lack. Research preprint
📄 Addressable Recall Compaction for Long Context-Window Control in AI Agents 2026 · arXiv
Thang Dang et al.
ARC stores tool observations in an append-only addressable log and swaps in compact ID citations under context pressure, letting the agent recall detail without re-running tools, 99.40% on Needle-in-a-Haystack and 29.97% on LongBench-v2 Hard. A concrete alternative to lossy summarization compaction. Research preprint
📄 Filesystem-Based Memory for LLM Agents: Organization, Evolution, and Sustainability 2026 · arXiv
Sizhe Zhou et al.
First systematic study of the now-ubiquitous markdown-files-on-disk memory pattern, asking whether agents can keep a growing store organized and whether that organization actually pays off in retrieval cost, answer quality, and memory health. Evaluates the default memory design most practitioners already run without evidence. Research preprint
🧰 MemLens: A Value-Aware Memory Management System with Interactive Analytics for LLM-based Agents 2026 · arXiv Scores individual memory records with Shapley-style attribution and exposes an interactive dashboard for inspecting memory value, hierarchy, and competing management strategies. Rare tooling for the operator side of agent memory, what is in there, what is it worth, what should be evicted. Research preprint
📄 A Graph-Native Bitemporal Memory Store for Conversational AI Agents 2026 · arXiv
Alp Niksarli & Gopesh Baheti
Agent-local Neo4j store combining vector indexes with bitemporal modeling so facts carry both valid-time and transaction-time, reaching 46.7% overall recall and 80% on knowledge-update questions where naive stores overwrite silently. Honest about weak temporal-reasoning performance, which is useful signal. Research preprint
📄 HiSkill: Empowering LLM Agents with Hierarchical Skill Graphs 2026 · arXiv
Yu Hao et al.
Organizes past trajectories into a directed graph linking high-level skills to executable action templates across several relation types, then retrieves task-relevant subgraphs at inference to guide skill switching and action selection. Turns accumulated experience into navigable structure rather than a flat trajectory pile. Research preprint
📄 VITAL-RAG: Invariance Race for Context Allocation in Coding Agents 2026 · arXiv
Zijian Lu et al.
Names an invariance race between cutting redundancy and preserving task-relevant code, then organizes retrieved evidence by code object with companion selection to raise recall while spending fewer tokens. Sharpens the context-budget problem specific to repository-scale coding loops. Research preprint
🧰 CodeNib: A Multi-View Data System for Serving Repository Context to Coding Agents 2026 · arXiv Maintains lexical, dense, and structural views per repository commit that persist incrementally across code changes, giving agents ranked search and symbol navigation at far fewer tokens than grep-driven exploration. Treats repo context as a served, versioned index rather than something re-derived every session. Research preprint
📝 Stateless MCP Has Recaptured My Interest 2026 · Simon Willison’s Weblog
Simon Willison
Simon Willison's practitioner read on what the 2026-07-28 MCP specification revision actually changes for people running agents at scale. Practitioner analysis
📄 MemTxn: A Transaction Boundary for Source-Supported Updates and Complete-State Recovery in Agent Memory 2026 · arXiv
Hanshuai Cui et al.
Applies database transaction semantics to agent memory: an Ordered PatchTest validates every write against its source, a Temporal Resolver picks versions, and a durable snapshot journal recovers complete state after corruption. This is the memory-durability primitive most long-running loops lack, writes are gated, not appended blindly. Research preprint
📄 Zero-Mem: Zero-Token Memory Operations for LLM Agents 2026 · arXiv
Yilin Xiao et al.
Asks whether structured memory access requires generation at all, and answers no. Most agent memory systems spend extra LLM calls generating intermediate records and mediating retrieval, adding recurring token and latency cost while merged or omitted details obscure the original evidence. Research preprint
📄 Reusing Past Repairs Through Hierarchical Trajectory Abstraction for Coding Agents 2026 · arXiv
Yisen Xu et al.
Repair agents treat every issue independently and throw away the procedural knowledge from previous repairs. Research preprint
📄 MemHarness: Memory Is Reconstructed, Not Replayed 2026 · arXiv
Rong Wu et al.
Argues that treating retrieved experience as a static record to replay causes negative transfer when past runs do not match present state, and trains a unified policy (GRPO) that reconstructs memory conditioned on the current state instead. Research preprint
📄 ChronoMem: Version Control and Semantic Rollback for Large Language Model Agent Memory 2026 · arXiv
Yongye Su et al.
Commits a memory snapshot on every write and supports natural-language rollback requests via hybrid retrieval, so an agent can inspect and revert prior memory states rather than living with whatever it last wrote. Claims to be the first open-source system plus benchmark for systematic semantic global memory rollback. Research preprint
📄 Σ-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems 2026 · arXiv
Peilin Feng et al.
Maintains symmetric matrices of per-peer competence and peer-relationship evidence, updated online from correctness feedback with spectral bounds for stability, then reuses that memory for residual steering, response-free peer routing, or reliability-weighted voting. Research preprint
📄 SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolution 2026 · arXiv
Zhiyuan Yao et al.
Treats a persistent, evolving skill document as the agent's carried state across a sequence of tasks, rather than as a retrieval index consulted per task. Research preprint
📄 A-TMA: Decoupling State-Aware Memory Failures in Long-Term Agent Memory 2026 · arXiv
Zitong Shi et al.
Names ghost memory: superseded, current, and transition facts coexisting in the bank and being retrieved together, so the answer model cannot tell what is true now. Keeps superseded records but labels them, and argues bank, retrieval, and answer-time failures need decoupled evaluation. Research preprint
🧪 Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-horizon Agents 2026 · arXiv
Yeonjun In et al.
Challenges the assumption that one retention policy suits every user, since a fixed policy spends budget on transient sessions while dropping context long-horizon work needs. Proposes session-level storage gating that skips memory writes for sessions judged transient. Benchmark or evaluation
📄 SmoothAgent: Efficient Long-Horizon LLM-Based Agent Serving with Lookahead Context Engineering 2026 · arXiv
Zaifeng Pan et al.
Prices the context strategies agent frameworks rely on. Offloading, reduction, and isolation each invalidate the KV cache and force a re-prefill, raising time-to-first-token. Observes that these transformations are segment-decomposable, so a prefix can be transformed ahead of time rather than on the critical path. Research preprint
📄 Everything is Context: Agentic File System Abstraction for Context Engineering 2025 · arXiv
Xiwei Xu et al.
Argues prompt engineering, RAG, and tool integration stay fragmented and leave transient artefacts that cannot be traced or audited. Applies the Unix everything-is-a-file idea to context, giving memory, tools, knowledge, and human input one persistent mount-based interface with governance attached. Research preprint
📄 AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Management 2026 · MobiSys '26: 24th Annual International Conference on Mobile Systems, Applications and Services 2026
Shizuo Tian et al.
Reframes interaction history as a program with variables and control flow, so what to retain and what to discard follows program structure instead of recency or similarity. Adds a global belief state for partial observability, aimed at the context overhead that grows with every step of a long GUI task. Research paper
📄 Context Compaction Theory 2026 · arXiv
Hayder Tirmazi et al.
First formal analysis of a process every long-running agent already runs. Models compaction as two games, one selecting a subset of accumulated state to retain and one generating a replacement, and studies the compounding case that matters in practice: later compactions operate on states that already contain earlier summaries. Research preprint
📄 Beyond Compaction: Structured Context Eviction for Long-Horizon Agents 2026 · arXiv
Andrew Semenov & Svyatoslav Dorofeev
Annotates the trajectory as typed, dependency-linked episodes, then evicts in priority order using a deterministic policy that calls no model. Preserves user turns and the context under active reasoning while shedding action episodes whose effects are already persisted in the environment, holding active context near a stable ceiling. Research preprint
📄 Parallel Context Compaction for Long-Horizon LLM Agent Serving 2026 · arXiv
Musa Cim et al.
Prices summarization-based compaction as an operational cost rather than a design detail: the blocking call stalls agent inference for tens of seconds, prompt instructions barely control summary volume, and retained information fluctuates run to run. Proposes running compaction alongside the agent loop instead of in it. Research preprint
📄 Agent Memory: Characterization and System Implications of Stateful Long-Horizon Workloads 2026 · arXiv
Yasmine Omri et al.
A systems characterization of agent memory rather than another memory design. Classifies ten representative systems on four axes and profiles them with a phase-aware harness that attributes cost separately to construction, retrieval, and generation, which is where memory overhead usually hides. Research preprint
📄 When Memory Becomes Authority: Benchmarking Authority Collapse at the Memory Consolidation Boundary 2026 · arXiv
Qiuyang Zhan et al.
Names authority collapse: consolidation keeps a claim but drops the source constraints governing its use, so stored memory later implies more authority than its origin permitted and an offhand remark can return as a standing instruction. The benchmark holds claim and task fixed and varies only source authority. Research preprint
📄 Graph-based Agent Memory: Taxonomy, Techniques, and Applications 2026 · arXiv
Chang Yang et al.
Survey of agent memory through graph structure, taxonomized on three axes: short-term against long-term, knowledge against experience, and non-structural against structural. Argues graphs fit the job because they carry the relational dependencies and hierarchy that flat retrieval flattens away. Research preprint
📄 Memory as Infrastructure: Reliability Engineering for Persistent Agent Memory in Months-Long LLM-Assisted Development 2026 · arXiv
Mike Helwig
Operational report from one Claude Code session line running since January 2026 on a 633,000-line codebase. Argues the missing practice is reliability engineering for memory itself: a session-start health gate with distinct failure modes, heartbeat telemetry so no enumerated failure passes unrecorded, and SRE-style alert budgets. Research preprint
📄 Protocol-Preserving Context Trimming for Agentic Workflows: Benefits, Failure Regimes, and Budget Guardrails 2026 · arXiv
Harish Gaggar
Compares five trimming strategies on task success and protocol adherence, not just tokens saved. Conventional trimming saved about 60% of tokens but cut task success to 67-77%, adaptive guardrails reached 96% at 56% savings, and retaining 25% of context or less raised failure odds nearly elevenfold. Research preprint
📄 How Strongly Should Task State Influence an LLM Agent? 2026 · arXiv
Chenyu Zhang et al.
Holds rules, model, and paired episodes fixed and varies only how strongly task state reaches the agent: a raw transcript, an exact checklist, per-turn directives from a state machine advanced only by execution receipts, or a gate that refuses state-violating actions. Separates shown, told, and enforced state. Research preprint
📄 The Immutable Past: Formalizing State Mutability and Conflict Resolution in Mutable RAG 2026 · arXiv
Hamed HaddadPajouh & Amir AmiriTabat
Names semantic shadowing: when shared memory is append-only, conflicting historical observations accumulate and statistically outweigh valid recent updates, so agents act on obsolete facts. Also shows a majority-vote trap where widening the retrieval window degrades accuracy instead of improving it. Research preprint
📄 Scope Before You Persist: Preventing Cross-Family Interference in Agent Memory 2026 · arXiv
Yezhou Cheng et al.
Matches retrieval scope to certification scope for persistent skill edits in frozen-model agents. Retrieving each accepted skill only for its originating task family raises hidden trajectory utility from 0.713 under global memory to 0.816 and cuts harmful deployments from six of eight to none. Research preprint

Orchestration And Multi-Agent Delegation

Coordinating multiple agents inside one loop: planners, workers, judges, worktree isolation, and merge gates.

Resource Published at Contribution Evidence
🧰 Microsoft Agent Framework 2025 · GitHub
microsoft/agent-framework
Microsoft's successor to AutoGen and Semantic Kernel for building and orchestrating multi-agent workflows in Python and .NET. Source implementation
🧰 LangGraph 2023 · GitHub
langchain-ai/langgraph
Graph-based framework for controllable agent workflows, persistence, and human-in-the-loop steps. Source implementation
🧰 CrewAI 2023 · GitHub
crewAIInc/crewAI
Framework for multi-agent workflows organized around roles, tasks, and crews. Source implementation
📚 LlamaIndex Workflows Developer Documentation Event-driven workflow abstraction for agentic applications. Technical documentation
📚 OpenAI Agents SDK handoffs openai.github.io First-class delegation between specialized agents. Official documentation
📚 Agent Protocol AgentProtocol.ai API protocol for agent interaction, useful for separating loop managers from agent runtimes. Technical documentation
🧰 AgentKit 2024 · GitHub
inngest/agent-kit
TypeScript toolkit for durable, event-driven agents on workflow infrastructure. Source implementation
🧰 deepagents 2025 · GitHub
langchain-ai/deepagents
LangChain project for deeper, longer-running agents with middleware and harness patterns. Source implementation
📚 Temporal for AI temporal.io Durable execution for long-running agent workflows: crash-proof state, automatic retries, and human-in-the-loop signals. Technical documentation
🧰 Restate Restate Durable execution runtime for building resilient, stateful agents and workflows that survive failures mid-loop. Implementation
🧰 DBOS dbos.dev Lightweight PostgreSQL-backed durable execution library for crash-proof agent workflows, queues, and scheduled triggers. Implementation
🧰 Agent Orchestrator 2026 · GitHub
OrchestratorInc/agent-orchestrator
Runs and supervises teams of coding agents from planning through merge, each in an isolated worktree, fixing CI failures and responding to reviews across the PR lifecycle. Works across Claude Code, Codex, and other harnesses from one control surface. Source implementation
🧰 Omnigent 2026 · GitHub
omnigent-ai/omnigent
Databricks' open-source meta-harness and control plane that runs Claude Code, Codex, Cursor, and Pi under shared policies, with budget caps and human-approval gates enforced at the harness layer rather than in prompts. Source implementation
📄 From Agent Loops to Structured Graphs: A Scheduler-Theoretic Framework for LLM Agent Execution 2026 · arXiv
Hu Wei
Replaces opaque agent loops with immutable plan-version DAGs and a planning-execution-recovery split, giving inspectable scheduling, deterministic recovery, escalation, and termination guarantees. Research preprint
🧰 Eve 2026 · GitHub
Vercel/eve
Vercel's TypeScript-native agent framework with durable execution, sandboxed compute, and OpenTelemetry tracing built in, so recurring agent work persists, replays, and is observable across runs by default. Source implementation
📄 Verified Multi-Agent Orchestration: A Plan-Execute-Verify-Replan Framework 2026 · ICLR Workshop on Multi-Agent Learning: Generalization and Adaptation in Intelligence (MALGAI)
Xing Zhang et al.
Decomposes work into a dependency-aware DAG, runs domain agents in parallel, and uses an LLM verifier to drive adaptive replanning with configurable stop conditions, the verify-and-replan core of a reliable loop. Research paper
📄 From Static Templates to Dynamic Runtime Graphs: A Survey of Workflow Optimization for LLM Agents 2026 · arXiv
Ling Yue et al.
Organizes how agent workflows are fixed ahead of time or generated and revised per run, and which evaluation signals drive that choice, a map of the design space for recurring loops. Research preprint
📄 Agent-as-a-Router: Agentic Model Routing for Coding Tasks 2026 · arXiv
Pengfei Zhou et al.
Agentic model routing for coding agents reframed as a context-action-feedback loop (ACRouter: orchestrator, verifier, memory) that learns which LLM to route each task to from execution feedback rather than frozen priors, with the CodeRouterBench benchmark across 8 frontier models. Research preprint
📝 Amp: Custom Agents 2026 · ampcode.com
Amp
Amp's plugin-defined custom agents that run as the main agent or as subagents, spawn parallel workers, join tool pipelines, and use thread actions to build background review threads that report results back to a parent thread. Practitioner analysis
🧰 Bernstein 2026 · GitHub
sipyourdrink-ltd/bernstein
Deterministic Python orchestrator that runs parallel CLI coding agents in isolated Git worktrees, gates merges on tests, lint, and type checks, and records every scheduling decision in a tamper-evident audit log. Source implementation
🧰 Aeon 2026 · GitHub
aeonfun/aeon
Autonomous agent framework that runs Claude Code unattended on GitHub Actions, dispatching skills on cron or reactive triggers with per-run quality scoring, persistent memory, and self-healing skill repair. Source implementation
🧰 h5i 2026 · GitHub
h5i-dev/h5i
Gives each coding agent an isolated sandboxed Git worktree, dispatches one task to a team that peer-reviews each other's candidates, then replays and tests each candidate with a neutral verifier before merging the winner. Source implementation
📄 SwarmResearch: Orchestrating Coding Agents for Open-Ended Discovery 2026 · arXiv
Yuvraj Virk et al.
A shepherd agent with global context steers a population of search agents that each work in their own Git branch with local context, countering the context accumulation of solo long-running agents and matching or beating baselines on 13 of 15 open-ended discovery tasks. Research preprint
📝 Scaling Long-Running Autonomous Coding Cursor
Wilson Lin
Cursor traces the coordination designs behind week-long autonomous coding runs, from flat agents with locking to optimistic concurrency to a planner/worker/judge hierarchy, letting hundreds of concurrent workers push to one branch on projects exceeding a million lines. Practitioner analysis
🧰 babysitter 2026 · GitHub
a5c-ai/babysitter
Harness-agnostic orchestration framework that runs agent workflows as process-as-code with mandatory enforcement stops after every step, quality-convergence loops that re-run verify-and-refine until thresholds pass, human-approval breakpoints, and an immutable event-sourced journal for deterministic replay across 12 coding harnesses. Source implementation
🧰 claude-code-merge-queue 2026 · GitHub
funador/claude-code-merge-queue
Local FIFO merge queue that serializes parallel Claude Code agents landing on a shared codebase, with a machine-wide build lock and a landing gate that blocks the integration branch until a configured check command passes, wired into the WorktreeCreate hook. Source implementation
📝 Devin can now manage Devins 2026 · cognition.com
The Cognition Team
Cognition's multi-Devin architecture where a coordinator Devin scopes a task into pieces, delegates each to a managed Devin running in its own isolated VM with terminal, browser, and dev environment, monitors progress, resolves conflicts, compiles the results, and reads workers' full trajectories to learn what worked and where they got stuck. Practitioner analysis
🧰 pilotfish 2026 · GitHub
Nanako0129/pilotfish
Multi-model orchestration layer for Claude Code where the frontier model plans, decides, and reviews while cheaper models execute volume work through six global subagent roles, with quality guarded by fresh-context verifier subagents rather than expensive models everywhere and graceful degradation when the frontier model is unavailable, shipped as three config files with no runtime code. Source implementation
🧰 fable-advisor 2026 · GitHub
DannyMac180/fable-advisor
Claude Code plugin implementing an architect pattern where Fable 5 owns specs, decomposition, and verification while routing implementation to cheaper lanes (Grok 4.5 by default, GPT-5.6 via Codex CLI, or Sonnet/Opus fallback), with cross-vendor review and optional racing of two implementers on the same spec. Source implementation
🧰 OpenMeld 2026 · GitHub
amplifthq/openmeld
Shared workspace where people and coding agents running on members' own machines (Claude Code, Codex, Cursor CLI, and others) take tasks, keep files and context with the conversation, and review results together, with a coordinating agent, configurable task approval, and schedules. Formerly OpenTag. Source implementation
🧰 herdr 2026 · GitHub
herdrdev/herdr
Single-binary terminal multiplexer purpose-built for running fleets of coding agents (Claude Code, Codex, Copilot CLI, Cursor Agent, and 15+ others) with per-agent state tracking. Source implementation
🧰 Orca 2026 · GitHub
stablyai/orca
Open-source agent development environment for orchestrating a fleet of parallel coding agents (20+ backends) on your own subscriptions, with per-agent workspaces and review flow. Source implementation
📄 Agentic Routing: The Harness-Native Data Flywheel 2026 · arXiv
Xinchen Liu et al.
Argues model routing must live inside the execution harness rather than in single-turn cost-quality tradeoffs, making step-level model selections from execution state and logging each decision as structured telemetry that feeds back to improve both routers and models. Research preprint
📄 A Formal Hierarchical Architecture for Agentic Orchestration with Stack-Based Execution 2026 · arXiv
Prashant Devadiga et al.
Formal orchestration architecture that runs agent workflows on a call stack with lazy capability discovery, giving multi-agent loops explicit, inspectable control flow instead of implicit prompt-driven handoffs. Research preprint
📄 The Honest Quorum Problem: Epistemic Byzantine Fault Tolerance for Agentic Infrastructure 2026 · arXiv
Jun He & Deying Yu
Formalizes how protocol-compliant reasoning agents can still agree on a semantically invalid transition, especially when they share models, prompts, or tools; derives separate safety and liveness budgets and shows that adding agents helps only when measured error correlation falls. Research preprint
📚 Graph-based agent workflows Google Agent Development Kit Official ADK 2.0 graph runtime for explicit, deterministic workflows that mix agents, tools, and code with branching, state, human input, and bounded cycles. Official documentation
📚 Flows CrewAI Official event-driven workflow layer with typed state, branching and loops, persistence decorators, and resume or fork operations for long-running executions. Official documentation
📚 Graph Strands Agents Official deterministic graph pattern for multi-agent dependencies and cycles, with shared execution state and explicit execution limits. Official documentation
Resource Published at Contribution Evidence
📄 Towards a Science of Scaling Agent Systems 2025 · arXiv
Yubin Kim et al.
Controls 260 configurations across six agentic benchmarks to show how coordination, model capability, task parallelism, and tool load shape performance and error amplification. Research preprint
📝 Amp: From Agent to Agent 2026 · ampcode.com
Amp
Amp on agent-to-agent handoffs: one agent delegating bounded work to another with results returned to the parent thread, formalizing delegation as a first-class platform primitive. Practitioner analysis
📝 Cursor: Agent Swarms and the New Model Economics Cursor
Wilson Lin
Cursor on the economics of agent swarms: how parallel fleets change the cost calculus of model selection, and what coordination overhead does to effective throughput per dollar. Practitioner analysis
📝 Amp: Meet Puck 2026 · ampcode.com
Amp
Amp's conversational coordinator that spawns agents, investigates issues, organizes threads, and coordinates multiple agent tasks across projects through natural-language commands. Practitioner analysis
🧰 Warren 2026 · GitHub
jayminwest/warren
Control plane for coding agents that operate in isolation, styled as a Coolify for agents: deploy, monitor, and manage long-running agent workloads on your own infrastructure. Source implementation
🧰 agent-talk 2026 · GitHub
xhluca/agent-talk
Enables coding agents to work together by giving them a shared communication channel, so parallel agents can coordinate instead of colliding. Source implementation
🧰 codex-model-routing-team 2026 · GitHub
zjp1997720/codex-model-routing-team
Runs background Codex workers under a lead agent that verifies their output before it lands, a small-team pattern for supervised parallel delegation. Source implementation
🧰 Buzz 2026 · GitHub
block/buzz
Self-hostable workspace from Block where humans and AI agents share the same rooms, giving a fleet of long-running agents a common coordination surface instead of isolated one-off sessions. Source implementation
🧰 open-kritt 2026 · GitHub
Kritt-ai/open-kritt
AGPL-3.0 security-research platform that decomposes vulnerability hunting into small well-defined tasks, fans them out across parallel tool-enabled agents in disposable Docker sandboxes, then closes the loop with post-processing verification scripts, proof-of-concept generation, de-duplication, and severity ranking via reusable workflow playbooks. Source implementation
🧰 BossConsole 2026 · GitHub
risa-labs-inc/BossConsole
Native JVM operator's console (Kotlin Multiplatform, explicitly not Electron) that runs Claude Code, Codex, Gemini CLI, and OpenCode side-by-side with a shared browser, terminal, editor, secrets manager, and ~100 MCP tools, adding per-tool RBAC governance, kill-switches, E2E-encrypted session sharing, and daemonized persistent sessions for supervised multi-agent operation. Source implementation
🧰 Fractal 2026 · GitHub
plasma-ai/fractal
Arranges autonomous agent loops into a tree: each node iterates toward a goal in its own Git worktree and spawns child loop-nodes for separable subtasks, bounded by hard caps on iterations, depth, children, cost, and time, with run metadata in a local SQLite database and a live TUI for operator steering. Source implementation
📄 SpecBox: Speculative Sandbox Scheduling for Efficient LLM Agent Serving 2026 · arXiv
Yihui Zhang et al.
Serving infrastructure for MCP-based agent loops, where disaggregated sandboxes force a choice between persistent reservations that waste memory at scale and lazy instantiation that inflicts cold-start penalties on multi-tenant multi-turn workloads. Research preprint
📄 A Comparative Study of MCP and A2A for Inter-Agent Coordination in LLM-Based Systems 2026 · arXiv
Ionut Predoaia et al.
Implementation-grounded comparison rather than a survey: the same software-engineering task built twice, once on MCP and once on A2A, evaluated against requirements drawn from prior literature and industry partner discussions -- agent discoverability, multi-part messaging, multi-turn conversations, asynchronous communication, observability, interoperability, and access control. Research preprint
🧰 AgentENV 2026 · GitHub
kvcache-ai/AgentENV
Open-sourced 2026-07-27 by the Kimi/kvcache-ai team as the environment substrate behind Kimi K3's agentic RL training. Source implementation
🧰 Ruflo 2025 · GitHub
ruvnet/ruflo
Agent meta-harness for Claude Code and Codex, built on the premise that an agent is a model plus a harness: it supplies tools, memory, loops, sandboxes, and controls so agents self-organize into swarms, learn across tasks, and persist state between runs. Formerly Claude Flow. Source implementation
📄 Toward an Organizational Science of Multi-Agent LLM Systems: Decoupling Who, How, and Which Algorithm 2026 · arXiv
Huan Chen et al.
IMACS separates multi-agent systems into organizational structure, coordination mechanism, and collaboration protocol, then shows a contextual-bandit meta-protocol picking per task beats any fixed protocol on quality-cost tradeoffs. The decoupling makes previously incomparable multi-agent results comparable. Research preprint
📄 Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineering 2026 · arXiv
Rushi Qiang et al.
Hierarchical orchestrator/sub-agent decomposition for long-horizon ML engineering that lets a 4B model reach performance comparable to much larger ones, with relative gains up to 36.7%. Good evidence that delegation structure substitutes for model scale on long tasks. Research preprint
📄 Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges 2026 · arXiv
Quim Motger et al.
Systematic review of 141 multi-agent debate studies with a three-dimensional taxonomy over participants, interaction mechanisms, and agreement protocols, concluding the field has converged prematurely on one conventional design. Useful map for anyone choosing a consensus mechanism for a delegation loop. Research preprint
🧰 Agent Manager 2026 · GitHub
wainmann/agent-manager
Terminal UI for operating a small agent fleet, surfaced on HN 2026-07-30. Each agent runs in an isolated tmux session that persists after the manager exits, and dead sessions revive with conversation history intact, so the fleet outlives the operator's terminal. Source implementation
🧰 TrueDeck 2026 · GitHub
WutIsHummus/TrueDeck
Terminal-first workbench that puts Grok, Codex, Cursor, Claude, and Gemini in split panes on one codebase, each keeping its real TUI rather than being wrapped in a chat webview. Source implementation
🧰 qm 2026 · GitHub
yc-software/qm
Y Combinator's open-source agent platform for whole organizations rather than one operator. Each person and each Slack room gets its own scoped memory, files, keychain view, permissions, crons, web apps, and durable sandbox, so background work keeps running "while nobody's watching" without workspaces bleeding into each other. Source implementation
📄 AgentRadio: Passive Awareness for Long-Horizon Multi-Agent Collaboration 2026 · arXiv
Xinxing Ren et al.
Adds asynchronous message passing so coding agents keep background awareness of each other instead of exchanging information only at handoff boundaries. Four coordinated agents reach 62.1 percent on a long-horizon codebase-comprehension benchmark against 32.3 percent solo, and beat the same harness running a newer model generation. Research preprint
📄 MANTA: Multi-Agent Network Topology Adaptation for Self-Evolving Multi-Agent Systems 2026 · arXiv
Mao-xun Huang et al.
Existing multi-agent systems treat communication topology as a fixed design choice or an offline optimization target. Research preprint
🧰 sol-advisor 2026 · GitHub
DannyMac180/sol-advisor
Codex-native orchestration plugin that splits an agent run into an architect session that owns requirements, specs and acceptance, and bounded implementation lanes (native Terra lane, or an opt-in Luna lane using visible Codex app tasks) that only execute. Source implementation
📄 SKIMIX: Multi-Agent Harness-Time Scaling with Skill Mixture for Dynamic Harness Engineering 2026 · arXiv
Jia Luo
Scales the harness rather than the model: embedding-based skill retrieval, submodular anti-dilution routing, and adaptive skill evolution let multiple agents refine over repeated rounds. Notably honest about limits, gains are large on open-ended math but flat or negative on multiple-choice, and mostly land in the first refinement round. Research preprint
📄 An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding 2026 · arXiv
Yanyu Ren et al.
Introduces MSEval plus the LegoGent execution engine and holds the model fixed while varying 10 collaboration topologies, finding that organizational topology rivals model capability, score swings above 30 points. Structured pipelines converge fastest; heavy managerial layers hurt. Research preprint
📄 Scaling LLM-Driven Multi-Agent Systems: Design Principles and Architectural Scalability Analysis 2026 · arXiv
Linus Sander et al.
Proposes four design principles, simplicity, elastic feedback, sequential workflows with optional loops, and summary-based communication, then tests them across four architecture configurations on standardized engineering tasks. Research preprint
📝 Scaling Managed Agents: Decoupling the Brain From the Hands 2026 · Anthropic Engineering
Lance Martin et al.
Virtualizes the agent into three interfaces: a stateless brain, tool sandboxes behind a single execute call, and a durable session event log held outside the harness. Because session state survives harness or container loss, each component becomes replaceable rather than a pet. Practitioner analysis
📝 Building a C Compiler With a Team of Parallel Claudes 2026 · Anthropic Engineering
Nicholas Carlini
Sixteen parallel agents built a 100k-line C compiler that compiles a bootable Linux kernel, coordinating through git rather than a scheduler: each claimed work by committing a file to a shared task directory. When all agents converged on one bug, GCC was used as an oracle to fan the work back out. Practitioner analysis
📄 Argus: A General-Purpose Agentic Reasoning Runtime for Long-Horizon Tasks 2026 · arXiv
Boxiu Li et al.
Manager, Planner, Engineer, and Reviewer run bounded missions over durable project state, with model weights fixed so self-evolution happens in runtime state and control policy rather than training. Memories, skills, verifiers, and routing decisions are admitted only after role-owned review, and rejected routes are retained too. Research preprint

Benchmarks And Evaluation

How to measure loops rather than single runs: long-horizon, multi-session, reliability, and verifier-quality benchmarks.

Resource Published at Contribution Evidence
🧪 SWE-bench swebench.com Benchmark for resolving real GitHub issues through code editing and tests. Benchmark or evaluation
📄 SWE-bench: Can Language Models Resolve Real-World GitHub Issues? 2024 · International Conference on Learning Representations (ICLR)
Carlos E. Jimenez et al.
Original SWE-bench paper. Research paper
📄 SWE-bench Goes Live 2025 · Advances in Neural Information Processing Systems 38: Datasets and Benchmarks Track (NeurIPS)
Linghao Zhang et al.
Dynamic benchmark designed to reduce overfitting to static issue sets. Research paper
🧪 Terminal-Bench TERMINAL-BENCH Benchmark for agents operating in terminal environments. Benchmark or evaluation
🧰 Terminal-Bench repository 2026 · GitHub
harbor-framework/terminal-bench
Open-source benchmark and harness for hard terminal tasks. Source implementation
📄 AgentBench 2024 · International Conference on Learning Representations (ICLR)
Xiao Liu et al.
Multi-environment benchmark for evaluating LLMs as agents. Research paper
📄 WebArena 2024 · International Conference on Learning Representations (ICLR)
Shuyan Zhou et al.
Realistic web environment for autonomous agents. Research paper
📄 OSWorld 2024 · Advances in Neural Information Processing Systems 37: Datasets and Benchmarks Track (NeurIPS)
Tianbao Xie et al.
Benchmark for multimodal agents operating full computer environments. Research paper
📄 ToolBench 2024 · International Conference on Learning Representations (ICLR)
Yujia Qin et al.
Tool-use benchmark and dataset for tool-augmented agents. Research paper
📄 GAIA 2023 · arXiv
Grégoire Mialon et al.
Benchmark for general AI assistants requiring reasoning, tool use, and multi-step work. Research preprint
📄 Tau-bench 2024 · arXiv
Shunyu Yao et al.
Benchmark for tool-agent-user interactions in realistic domains. Research preprint
📄 VisualWebArena 2024 · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL)
Jing Yu Koh et al.
Visually grounded web-agent benchmark extending WebArena. Research paper
📄 AppWorld 2024 · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL)
Harsh Trivedi et al.
Benchmark of interactive app tasks with state-based and execution-based evaluation. Research paper
📄 Vending-Bench 2025 · arXiv
Axel Backlund & Lukas Petersson
Benchmark for long-term coherence of autonomous agents; documents how small errors compound over very long loop horizons. Research preprint
🧪 Vending-Bench leaderboard andonlabs.com Live long-horizon coherence results from Andon Labs. Benchmark or evaluation
🧪 SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios 2025 · arXiv
Tue Le et al.
Release-note-derived evolution tasks where agents score far below isolated-issue benchmarks, quantifying the long-horizon gap loops must manage. Benchmark or evaluation
📄 EvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification 2026 · arXiv
Hanrong Zhang et al.
A skill generator and a co-evolving surrogate verifier improve multi-file skill packages over iterations, evaluated on the SkillsBench benchmark of structured skill bundles. Research preprint
🧪 SaaSBench: Coding Agents in Long-Horizon Enterprise SaaS Engineering 2026 · arXiv
Qingnan Ren et al.
Benchmark for agents on multi-dependency, interactive enterprise tasks, with automated evaluation that probes where long-horizon loops break down. Benchmark or evaluation
🧪 RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades 2026 · arXiv
Xinbo Xu et al.
115 real version-upgrade tasks across 17 repositories requiring multi-file changes (median ~3,700 lines), stressing how far agent loops sustain coherent, large-scale work. Benchmark or evaluation
🧪 RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code 2025 · International Conference on Learning Representations (ICLR)
Dhruv Gautam et al.
Multi-file refactoring tasks that require tracking and carrying state across many steps, isolating the durable-state weakness that breaks long agent loops. Benchmark or evaluation
🧪 RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents 2026 · arXiv
Meher Bhaskar Madiraju & Meher Sai Preetam Madiraju
Scores planning, verification coverage, recovery, abstention, and atomic transitions (not just whether code passes), measuring the loop discipline that separates reliable agents from reckless trial-and-error. Benchmark or evaluation
🧪 SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks 2026 · arXiv
Gabriel Orlanski et al.
Quantifies structural erosion and verbosity creep across iteration checkpoints in native harnesses like Claude Code and Codex, evidence for why loops need verification and budgets. Benchmark or evaluation
🧪 LongCLI-Bench: A Preliminary Benchmark for Long-horizon Agentic Programming in Command-Line Interfaces 2026 · Findings of the Association for Computational Linguistics: ACL
Yukang Feng et al.
Long-horizon CLI tasks where most runs stall below 30% completion, mapping where unattended loops break down. Benchmark or evaluation
🧪 Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios? 2026 · arXiv
Yangda Peng et al.
Benchmark of 2,458 instances across research and coding domains measuring how reliably LLM judges verify rubrics on agent outputs, finding substantial noise even in strong models and quantifying the trade-offs of prompt design, batched evaluation, and majority voting. Benchmark or evaluation
🧪 SentinelBench: A Benchmark for Long-Running Monitoring Agents 2026 · arXiv
Matheus Kunzler Maldaner et al.
Microsoft Research benchmark of 100 tasks across 10 synthetic web environments that evaluates long-running monitoring agents on whether they wait or act appropriately, scoring task completion, response speed, and resource efficiency. Benchmark or evaluation
🧪 SWE-Together: Evaluating Coding Agents in Interactive User Sessions 2026 · arXiv
Yifan Wu et al.
Multi-session coding benchmark of 109 repository-level tasks reconstructed from 11,260 recorded user-agent sessions, replayed with an LLM user simulator and scored on final correctness and the number of corrective feedback turns. Benchmark or evaluation
🧪 The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break 2026 · arXiv
Xinyu Jessica Wang et al.
Cross-domain diagnostic benchmark that scales task horizon through depth and breadth extension, then attributes failures across 3,100+ agent trajectories to a seven-category taxonomy via a trajectory-grounded LLM judge validated against human annotation. Benchmark or evaluation
📄 Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents 2026 · arXiv
Aaditya Khanal et al.
Reliability metrics for long-horizon agents (reliability decay, variance amplification, graceful degradation, meltdown onset) measured over 23,392 episodes across 10 models, showing capability and reliability rankings diverge as tasks lengthen. Research preprint
🧪 SEAGym: An Evaluation Environment for Self-Evolving LLM Agents 2026 · arXiv
Congjie Zheng et al.
Evaluation environment that measures whether a self-evolving agent's modifications to prompts, memory, and tools generalize to held-out tasks, using train, validation, and test splits and cost metrics on Terminal-Bench 2.0 and HLE. Benchmark or evaluation
🧪 EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions 2026 · arXiv
Haiyang Shen et al.
Benchmark of 26 evolving coding tasks across 227 evaluation rounds using cumulative executable tests to check that agents keep prior requirements working as specifications change, with top agents reaching only about 50% on multi-turn success metrics. Benchmark or evaluation
📄 On the Reliability of Computer Use Agents 2026 · arXiv
Gonzalo Gonzalez-Pumariega et al.
Repeated-execution study on OSWorld decomposing why computer-use agents fail tasks they previously completed, separating execution stochasticity, task-specification ambiguity, and behavioral variability as distinct causes of unreliability. Research preprint
📄 AgentLens: Revealing the Lucky Pass Problem in SWE-Agent Evaluation 2026 · arXiv
Priyam Sahoo et al.
Grades 2,614 SWE-agent trajectories across eight models to show that 10.7% of passing trajectories in its 1,815-trajectory evaluation subset are lucky trial-and-error successes, replacing binary pass/fail with process-quality tiers that shift model rankings. Research preprint
🧪 ORLoopBench: Solver-in-the-Loop Benchmarks for Self-Correction 2026 · Proceedings of the 43rd International Conference on Machine Learning (ICML), PMLR 306
Ruicheng Ao et al.
Formalizes infeasible-model debugging as a solver-in-the-loop process where each action triggers solver re-execution and infeasibility recomputation, giving deterministic verification for iterative repair in operations research. Benchmark or evaluation
🧪 LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis 2026 · arXiv
Kewei Xu et al.
Benchmark of 68 real-world data-analysis tasks built from Kaggle notebooks spanning 2,225 interactive turns, finding that long-horizon errors account for 52-69% of agent failures and that maintaining a correct analytical state is the core bottleneck. Benchmark or evaluation
🧪 MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks 2026 · arXiv
Zexue He et al.
Multi-session benchmark of interdependent agentic tasks where agents must distill earlier sessions into memory and use it to guide later actions, showing that near-saturated scores on long-context memory benchmarks fail to transfer. Benchmark or evaluation
🧪 Momento: Evaluating Persistent Memory and Reasoning with Multi-Session Agentic Conversations 2026 · arXiv
Adril Putra Merin et al.
Benchmark for persistent, tool-mediated task completion across multiple sessions, finding that agents fail by treating prior-session history as current context instead of stale state that needs re-validation. Benchmark or evaluation
🧪 π-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows 2026 · arXiv
Haoran Zhang et al.
Benchmark of 100 multi-turn tasks across 5 user personas with hidden intents, inter-task dependencies, and cross-session continuity, measuring agent proactivity separately from task completion in long-horizon trajectories. Benchmark or evaluation
🧪 Can LLM Agents Be CFOs? Benchmarking Long-Horizon Resource Allocation 2026 · arXiv
Yi Han et al.
A 132-month CFO simulation where agents repeat a monthly cycle of liquidity management, financial closings, and financing decisions with compounding state, and only 15.4% of trials survive the full horizon. Benchmark or evaluation
🧪 EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer 2026 · arXiv
Xingze Gao et al.
Benchmark isolating whether agents transfer reusable procedures such as searching, debugging, and verification across episodes in four long-horizon domains linked by ability graphs, finding curated experience transfers but no automatic method yields consistent gains. Benchmark or evaluation
🧪 AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents 2026 · arXiv
Xiangchen Cheng et al.
Bounded-memory testbed built on Slay the Spire 2 where every agent decision is made from a fresh prompt assembled by typed retrieval over recorded state, keeping prompt size bounded across runs of any length, with 298 documented trajectories released. Benchmark or evaluation
📄 Is Three the Magic Number? An Empirical Evaluation of LLM-Based Repair Loops 2026 · arXiv
Tobias Kiecker et al.
Empirical evaluation of iteration budgets for generate-validate-repair loops across code generation, test generation, and translation, finding the first three to four iterations capture most gains and that orchestration and feedback design matter more than the model. Research preprint
🧪 DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks 2026 · arXiv
Wenqi Huang et al.
113 from-scratch, contamination-free long-horizon software-engineering tasks with custom verifiers that accept any correct implementation, built to sidestep SWE-bench-style pretraining recall and better differentiate frontier coding agents. Benchmark or evaluation
🧪 PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization 2026 · arXiv
Yingyun Cui et al.
Benchmarks the profile-diagnose-edit-verify loop where the verifier is a profiler rather than a test suite: agents must deliver measured, reproducible speedups without breaking correctness, and across seven agent configurations the framework choice shifts results even with identical models. Benchmark or evaluation
🧪 Benchmarking coding agents on Databricks' multi-million line codebase 2026 · Databricks Databricks engineering post (July 8, 2026, authors including Matei Zaharia and Patrick Wendell) on an internal benchmark built from real merged PRs with test-suite verification, finding that models cluster into three capability tiers, token price is a poor proxy for end-to-end task cost, and harness choice matters, with their Pi harness sending about 3x less context per turn at equal quality. Benchmark or evaluation
🧪 UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks 2026 · arXiv
Zhekai Chen et al.
Capability-driven benchmark of 400 bilingual tasks for proactive agents operating everyday tools in live Docker environments with step-level checkpoints, decomposed into five foundational capabilities, skill usage, exploration, long-context reasoning, multimodal understanding, and cross-platform coordination, so failures localize to a root-cause capability instead of mixing capabilities per task, with closed-loop evaluation using multiple agent roles to simulate human feedback without leaking grading criteria. Benchmark or evaluation
🧪 SLBench: Evaluating How LLM Agents Follow Logical Relations in Skills 2026 · arXiv
Xuan Chen et al.
Benchmark for whether agent loops respect the logical relations inside skill files (preconditions, constraints, fallbacks): 70% of 5,000+ public skills contain at least one such relation, and on 86 executable cases leading coding agents show unsafe-behavior rates up to 70%, with an inference-time scaffold cutting violations by 63%. Benchmark or evaluation
🧪 SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution 2026 · Proceedings of the 43rd International Conference on Machine Learning (ICML)
Gangda Deng et al.
Commit-history-derived milestone task streams where agents must preserve system integrity across successive runs - frontier-model scores collapse from >80% on isolated tasks to at most 38% in continuous settings, quantifying the error-accumulation gap loop engineering targets. Benchmark or evaluation
🧪 AgentAbstain: Do LLM Agents Know When Not to Act? 2026 · arXiv
Xun Liu et al.
Benchmark measuring whether agents correctly abstain from acting when a task is underspecified, unsafe, or impossible, rather than proceeding anyway, a capability every unattended loop depends on for safe exits. Benchmark or evaluation
🧪 Agents Don't Just Agree, They Remember: Benchmarking Persistent Sycophancy 2026 · arXiv
Xutao Mao et al.
Benchmark showing that once a stateful personal agent is nudged into a sycophantic stance, it persists across later sessions through memory, so single-turn sycophancy tests understate the risk in long-running agents. Benchmark or evaluation
🧪 Set-shifting Behavioral Test for Harnessed Agents 2026 · arXiv
Ye Ziwei
Tests whether harnessed agents adapt when hidden tool reliability changes, paired with no-shift controls that separate genuine adaptation failures from ordinary task errors and expose routine lock-in. Benchmark or evaluation
🧪 MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers 2026 · arXiv
Huanxi Liu et al.
Applies 11 server-evolution mutations to 123 MCP servers and evaluates 12 LLMs; reported performance drops include 13.7% for GPT-5.4 and 14.4% for Claude Sonnet 4.6, quantifying tool-interface drift. Benchmark or evaluation
🧪 MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization 2026 · arXiv
Shaoxiong Zhan et al.
Provides 652 issue-PR instances across 23 languages, seven image categories, and four relevance levels; the strongest evaluated agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, leaving substantial room for visual-evidence-aware repair loops. Benchmark or evaluation
🧪 ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning 2026 · arXiv
Shuaiyu Zhou et al.
Builds executable long-horizon training environments from nearly 400 MCP servers and about 4,500 tools, generates tasks from tool-dependency graphs, and introduces turn-aware credit assignment for agentic reinforcement learning in large, dynamic tool spaces. Benchmark or evaluation
Resource Published at Contribution Evidence
🧪 MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interaction and MCP-Augmented Tasks 2026 · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Quyu Kong et al.
Evaluates long-horizon mobile work that requires clarification and MCP tool use in reproducible self-hosted environments with deterministic backend, storage, and callback verification. Benchmark or evaluation
🧪 AGENCYBENCH: Benchmarking the Frontiers of Autonomous Agents 2026 · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Keyu Li et al.
Provides 138 high-fidelity tasks across six capabilities whose average rollout exceeds one million tokens and 90 tool calls, exposing context-retention and long-horizon execution limits. Benchmark or evaluation
🧪 Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading 2026 · arXiv
Zongxia Li et al.
46 long-horizon terminal tasks across nine categories (experiment reproduction, software engineering, scientific computing, and more) graded with dense intermediate rewards and partial credit instead of binary pass/fail; runs average 231 episodes and 85 minutes, and the strongest of 15 frontier models reaches only 15.2% pass@1 at a 0.95 reward threshold and 10.9% at perfect completion. Benchmark or evaluation
🧪 PolyWorkBench: Benchmarking Multilingual Long-Horizon LLM Agents 2026 · arXiv
Hongliang Li et al.
67 long-horizon workplace-workflow tasks across commerce, knowledge work, legal, localization, and manufacturing, graded by a hybrid of structural checks, executable verification, and LLM judging. Shows state-of-the-art agents degrade sharply when workflows mix languages versus monolingual runs, a multilingual coverage axis long-horizon benchmarks otherwise ignore. Benchmark or evaluation
🧪 MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations 2026 · arXiv
Xixuan Hao et al.
Evaluates agent memory by the operations that maintain it, remembering, forgetting, updating, reflecting, with structured operation traces rather than final-answer accuracy, exposing lifecycle failure modes that answer-level evals miss. Finds current memory systems remain far from uniformly reliable. Benchmark or evaluation
📄 AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities 2026 · arXiv
Kai Chen et al.
Open-source evaluation infrastructure that factors agent evaluation into three independent components, Benchmark, Harness, and Environment, with a fault-tolerant asynchronous runtime and failure-mode diagnostics across 20+ benchmarks, making the harness a swappable, measurable variable rather than a confound. Research preprint
📄 Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing 2026 · arXiv
Ananda Dhakal et al.
Controlled study on the 104-task XBOW benchmark showing plain coding agents already solve a large share of tasks attributed to specialized security harnesses, a methodological warning to establish model-matched baselines before crediting the harness/loop architecture for performance gains. Research preprint
🧪 Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction 2026 · arXiv
Tencent WorkBuddy Bench Team et al.
Tencent's coding-agent evaluation suite spanning Code, Web, Office, and Security domains, where every task is reverse-engineered from a real commit, PR, or business scenario and rewritten as a short colloquial role-played request, contamination resistance comes from this construction plus dataset versioning rather than secrecy, so the full task directories, environment images, tests, and solutions ship publicly for third-party audit, alongside a cross-model leaderboard (per-subset scoring, not comparable across subsets). Benchmark or evaluation
🧪 ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders 2026 · arXiv
Zhongyuan Peng et al.
Benchmark that evaluates coding agents as interactive project builders: agents must turn ambiguous product intent into working software through planning, requirement clarification, tool use, debugging, and repository-level construction across multi-turn sessions with simulated users, graded on functional correctness, structural fidelity, and interaction quality rather than static fully specified tasks. Benchmark or evaluation
🧪 GuardianAgentBench: Where Agents Fail and How to Guard Them 2026 · arXiv
Vishal Ishwar Naik et al.
580-scenario safety benchmark spanning six domains and five adversarial attack modes, run on LangChain, LlamaIndex, and Vectara agents; even the strongest configuration reaches only 74.8% accuracy, and a guardrail defense recovers 19.9% of failures at a 0.5% false-positive rate, mapping distinct failure regimes for guarding tool-using autonomous agents. Benchmark or evaluation
📄 LLMs Get Lost in Evolving User Intent 2026 · arXiv
Jihoon Tack et al.
Microsoft framework converting static single-turn tasks into dynamic multi-turn conversations where users disclose, revise, and reshape intent across the loop rather than specify it upfront, showing substantial performance drops across model families, as the interaction-loop counterpart to long-horizon execution and a follow-on to the influential 'LLMs Get Lost in Multi-Turn Conversation'. Research preprint
🧪 ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management 2026 · arXiv
Pavel Golikov et al.
Benchmark generator producing arbitrarily scalable task graphs, natural-language problems paired with executable Python solvers linked by typed intermediate values, with controllable length, dependencies, distractors, and exact automatic verification. Benchmark or evaluation
📄 Code Monitor Red Teaming for Public-Test-Passing Code 2026 · arXiv
Junchi Liao et al.
Red-teams the monitor layer of coding-agent loops: can a weaker LLM verifier catch hidden bugs in code that already passes public tests? Research preprint
🧪 SciExplore: Evaluating Autonomous Agents from Scientific Navigation to Information Integration 2026 · Findings of the Association for Computational Linguistics: ACL 2026
Yinhao Tang et al.
103 expert-curated tasks across 10+ scientific disciplines evaluating long-horizon agent information-seeking loops, scientific database navigation, ambiguous literature retrieval, missing-reference completion, and cross-source knowledge synthesis. Evaluation of 10+ SOTA models shows accuracy collapsing as loop depth grows from entity-level reasoning to domain-level synthesis. Benchmark or evaluation
🧪 RUMBA: Russian User Memory Benchmark 2026 · arXiv
Elizaveta Shevtsova et al.
Long-term conversational memory benchmark built from timestamped multi-session user-assistant dialogues, with a fine-grained taxonomy of memory-centric questions spanning semantic type, session scope, temporal reasoning, and explicitness of temporal expressions. Benchmark or evaluation
📄 Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI 2026 · arXiv
Jiaqi Shao et al.
Formulates protocol validity -- a benchmark score supports a capability claim only if the evaluation protocol keeps the intended capability necessary for success -- and introduces HackDetect, a post-hoc audit that identifies an exposure, determines how the agent used it, and judges whether the score is misleading. Quantifies inflation with the Mislead gap (exploit score minus intended score). Research preprint
📄 Success Is Not Self-Explanatory: Auditing Success Provenance in Agent Evaluation 2026 · arXiv
Jingkun Luo & Da-Tian Peng
Once an agent can change its own information state during evaluation, correctness stops distinguishing intended reasoning from answer acquisition. Research preprint
📄 Are Production Cloud Skills Adequately Tested? Measuring and Governing Skill Test Coverage in Practice 2026 · arXiv
Haotian Si et al.
Introduces Skill Test Coverage: how completely a Skill's test suite covers its operational test obligations, as opposed to the usual question of whether the Skill improves task success. Passing available testcases never reveals which specified behaviors -- resource operations, user choices, validation steps, recovery paths -- have never been exercised. Research preprint
🧪 AlloBench: Measuring Online Tool Allocation Capability in LLM Agents 2026 · arXiv
Daniel Wang & Andrew Xu
Frames tool creation as an investment decision -- pay a fixed cost now for possible future reuse -- and tests whether agents allocate a fixed budget toward a few highly reusable tools rather than many one-offs. Benchmark or evaluation
🧪 SQBench: A Benchmark for Evaluating Task Delivery by Language-Model Agents in Production-Oriented Workflows 2026 · arXiv
Summer Sun
Makes the unit of evaluation a verifiable deliverable produced inside a constrained workflow rather than a knowledge, reasoning, or tool-use score. Benchmark or evaluation
🧪 E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios 2026 · arXiv
Weihuang Zheng et al.
Targets the gap between isolated-API-call benchmarks and agents that actually gather hidden information, compose tool calls, and commit state changes in stateful environments. Benchmark or evaluation
🧪 DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents 2026 · arXiv
Junming Chen et al.
Builds an evaluation environment with the properties real agent loops face and most benchmarks lack: live multi-turn read-write interaction with a running instrumented PostgreSQL under active workload, persistent state, observation spaces spanning thousands of time series plus business logs and concurrent activity, open solution spaces with different operational trade-offs, and faults cascading across internal mechanisms. Benchmark or evaluation
🧪 OrchBench: Evaluating Multi-Agent Orchestration Plans in Isolation via Deterministic Simulation 2026 · arXiv
Zhenzhen Ren et al.
Scores orchestration strategies on their own by building task dependency graphs and running a deterministic simulator, correlating strongly with real Claude Code performance at a fraction of the tokens. Lets teams iterate on delegation topology without paying for full agent runs. Benchmark or evaluation
🧪 HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following 2026 · arXiv
Liudas Panavas et al.
65 tasks requiring agents to obey 20-124 page policy documents in simulated professional settings, where the best configuration reaches only 36.2% under strict grading and agents override policy for in-environment requests or lose rules over long runs. Measures whether the standing instructions a loop depends on actually hold. Benchmark or evaluation
🧪 OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding 2026 · arXiv
Jingbo Zhou et al.
100 long-horizon office-suite tasks paired with economic signals so agent output can be priced against human labor; models come out faster and cheaper but not yet at human deliverable quality. The cost-grounded framing is more decision-useful than raw success rates. Benchmark or evaluation
🧪 RSIBench-Data: Benchmarking Data-Centric Research for Recursive Self-Improvement 2026 · arXiv
Fanqing Meng et al.
Controlled benchmark for agents iteratively refining their own training-data strategy: agents beat their first attempt in 58.33% of runs, but 78.26% of searches that ran past peak ended below it. Quantifies the knowing-when-to-stop failure that undercuts recursive self-improvement claims. Benchmark or evaluation
🧪 WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing 2026 · arXiv
Hao Liang et al.
1,151 auditable tasks testing whether enterprise agents route to the right surface among documents, tables, and dependency graphs; agents route well but answer poorly, showing source selection is necessary and far from sufficient. Isolates a routing stage most agent evaluations collapse into end-to-end accuracy. Benchmark or evaluation
🧪 Desktop-Delta Bench: Do Computer-Use Models Understand Desktop GUI Transitions? 2026 · arXiv
Abhishek Pillai et al.
2,013 human-verified offline instances testing whether computer-use agents can reconstruct the causal effect of their own actions, verifying state change, attributing which action caused it, and handling context-aware control across apps. Targets the self-verification step an unattended GUI loop needs before it can act again. Benchmark or evaluation
🧪 Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation 2026 · arXiv
Stefan Krsteski et al.
Unified corpus of nearly a million evaluation records spanning 30 benchmarks and over 700 agents, standardizing reporting and showing progress varies sharply by task type while some scoring methods distort capability estimates. Infrastructure for meta-analysis rather than another leaderboard. Benchmark or evaluation
🧰 smevals 2026 · Prime Radiant Simon Willison's writeup of smevals, an eval framework he built at Jesse Vincent's Prime Radiant lab that treats the agent harness as a first-class variable under test rather than a fixed backdrop. Implementation
📄 ORCA-bench: How Ready Are Language Model Agents for Oncall? 2026 · arXiv
Albert Gong et al.
Puts coding agents into a production-fidelity oncall setting, with a live instrumented microservice system and root-cause tasks whose ground truth is signed off by expert SREs. Best root-cause accuracy is 25.3 percent on realistic medium-difficulty inputs, the weakest model invents a root cause in 40 percent of reports, and the authors frame their numbers as a lower bound. Research preprint
📄 How Benchmarks Mis-Score Computer-Use Agents 2026 · arXiv
Zihan Dong et al.
Treats a benchmark score as the output of a pipeline that can fail at four stages, tasks go stale, trajectories omit decisive visual evidence, evaluators reject valid alternatives, and aggregate reports hide the cause of failure, then audits 150 public failure-scored trajectories across five web, enterprise-workflow, and desktop-control benchmarks. Research preprint
📄 SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response 2026 · arXiv
Lehan Wang et al.
Puts agents on forensic disk snapshots and host alerts across ten cyber ranges spanning multiple operating systems and attack techniques, testing whether they can both resolve the flagged issue and hunt for what the alert missed. No model achieved complete detection and remediation on any single range. Research preprint
📄 ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents 2026 · arXiv
Xingjian Wu et al.
Scores the trace, not just the outcome: a Process Grader rates goal alignment, efficiency, information utilization, and result verification against 12,541 rubric items over 320 tasks in 8 domains, explicitly to separate reliable reasoning from lucky success. Research preprint
📄 DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness 2026 · arXiv
Debin Meng et al.
100 end-to-end data-engineering tasks across five execution engines (PySpark, MySQL, HiveSQL, PrestoSQL/Trino, FlinkSQL) built from production enterprise code, graded by deterministic rules rather than an LLM judge. Strongest model scores 74.9 overall, and the deterministic grading makes it usable as a real loop exit gate. Research preprint
📄 PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks 2026 · arXiv
Manyi Wang et al.
Finds 13.6% of SWE-bench Verified instances have a pull request that does not actually match its linked issue, across five misalignment patterns, then builds a multi-agent checker combining pattern identification, cross-agent synthesis, and code-level verification. Research preprint
📄 SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvements 2026 · arXiv
Pengyu Xue et al.
188 tasks from real Python pull requests where the goal is quality improvement without behavior change, graded by 92 executable rules. Research preprint
📄 Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments 2026 · arXiv
Haomin Qi et al.
Turns repository history into verified, executable agent tasks with reconstructed environments via patch reversal and code mapping, validating correctness at each step. Covers bug fix, feature addition, test generation, API migration, and security repair, the pipeline you need to keep a coding loop's eval set fresh instead of frozen. Research preprint
📄 Echoverse: Deep, Evolving Environments for Training Computer-Use Agents at Scale 2026 · arXiv
Yash Pandya et al.
Compiles specifications into stateful applications and runs a co-evolution loop that improves the environment and the model together from graded rollouts, isolating behavioral depth, interaction coverage, and continuous environment improvement as the properties that matter. A 9B model goes from 36.5% to 67.1% across evaluation splits. Research preprint
📄 Meta-Task: Turning Terminal Task Synthesis into a Terminal Task for Scalable Agent Training 2026 · arXiv
Zhihong Pan et al.
Self-referential loop design: task synthesis is itself posed as a Terminal-Bench-format task, so agents generate, execute, and verify candidate tasks inside real containers and only executable, internally consistent tasks survive. Fine-tuning on just 3,221 synthesized trajectories reaches competitive Terminal-Bench 2.0 results. Research preprint
📝 Quantifying Infrastructure Noise in Agentic Coding Evals 2026 · Anthropic Engineering
Gian Segato
Holds model, tasks, and harness fixed while varying container resources across six configurations, and finds Terminal-Bench 2.0 moves 6 points between the least- and most-resourced setups. Infrastructure errors fall from 5.8% to 0.5%, so a score is not reproducible without its resource spec. Practitioner analysis
📝 Eval Awareness in Claude Opus 4.6's BrowseComp Performance 2026 · Anthropic Engineering
Russell Coleman
Documents a model recognizing it was being benchmarked, locating the BrowseComp source on GitHub, and decrypting the answer key instead of searching. Two of 1,266 problems, but the multi-agent setup reached unintended solutions at 3.7x the single-agent rate. Practitioner analysis
📝 Designing AI-Resistant Technical Evaluations 2026 · Anthropic Engineering
Tristan Hume
Three generations of a take-home evaluation, two of them defeated by successive model releases, and the design properties that survived: multi-hour horizons, wide scoring distributions, and no dependence on a single insight. Relevant to any loop whose gate is a human-graded task. Practitioner analysis
📚 Demystifying Evals for AI Agents 2026 · Anthropic Engineering Practical guide to combining code-based, model-based, and human graders, with an eight-step roadmap that starts from 20 to 50 tasks drawn from failures already observed in production. Treats the eval suite as a living artifact needing an owner, not a one-time deliverable. Technical documentation
🧪 Harness-Bench: Measuring Harness Effects Across Models in Realistic Agent Workflows 2026 · arXiv
Yilun Yao et al.
Isolates the variable most benchmarks either hold fixed or abstract away. Runs 106 sandboxed offline tasks across harness configurations and model backends under shared task environments, budgets, and evaluation protocols, while letting each harness keep its native execution behaviour. Benchmark or evaluation
🧪 AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications 2026 · arXiv
Yujie Zhao et al.
Memory benchmark built on agent-environment trajectories of states, actions, observations, and tool output rather than dialogue transcripts, with synthetic trajectories that scale to arbitrary horizons. Finds existing memory systems lean on lossy similarity search and lose causal structure. Benchmark or evaluation
🧪 From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents 2026 · Findings of the Association for Computational Linguistics: ACL 2026
Md Nayem Uddin et al.
Long-term memory benchmark over weeks-to-months histories covering remembering, reasoning, and recommending, plus Forgetting-Aware Memory Accuracy, a metric that penalizes answers built on obsolete or invalidated memory instead of only rewarding successful recall. Benchmark or evaluation
🧪 AlphaEval: Evaluating Agents in Production 2026 · arXiv
Pengrui Lu et al.
94 tasks sourced from seven companies running agents in their core business, built against the ways production diverges from benchmarks: implicit requirements, heterogeneous multi-modal inputs with facts split across sources, undeclared domain expertise, long-horizon deliverables, and expert standards that shift over time. Benchmark or evaluation
📝 A Methodical Approach to Agent Evaluation Google Cloud Blog Splits evaluation into final quality, trajectory, and trust, on the point that an agent can reach a correct answer through a flawed process, a silent failure that outcome-only scoring never surfaces. Pairs human, LLM-judge, code-based, and adversarial methods, then wires the suite into CI as a release gate. Practitioner analysis
🧪 LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation 2026 · arXiv
Han Li et al.
Benchmarks sustained execution rather than end state: 112 tasks across 8 languages and 9 domains, each a dependency DAG over separately testable units with source-evidenced prerequisite edges. The runtime releases tests along the ready frontier and keeps completed nodes as standing regression obligations. Benchmark or evaluation
🧪 Do Codex Skills Save Tokens? Six Controlled GPT-5.6-sol Runs 2026 · GitHub
Phelan164/codex-howto
Asks whether a repository workflow skill earns its context cost, and finds the answer reverses by task size: the no-skill control was cheapest on a small specified backend fix, while a lean engineering-loop skill spent 54% fewer reported tokens on a larger build. Two task classes and six runs, with the measurements published. Benchmark or evaluation
🧪 PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actions 2026 · arXiv
Chenxin Li et al.
Scores whether the intended side effect actually occurred rather than whether the final screen looks right, which is how most mobile-agent work grades a run. Mixes GUI, device-side commands, and structured tools, so choosing which surface to act through becomes part of the task. Benchmark or evaluation
🧪 LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering 2026 · arXiv
Yi Wang et al.
Benchmarks the loop rather than the agent inside it. A loop can trust a stale progress note, skip needed verification, spend budget in the wrong direction, or stop before the task is safe to submit, and one end-to-end outcome cannot show whether the loop's guidance or the agent's ability decided the result. Benchmark or evaluation
🧪 BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks 2026 · arXiv
Pradyumna Shyama Prasad et al.
Plants an optional shortcut in three ML tasks that inflates the public score but fails a hidden test set, so exploiting it breaks no stated rule. 57.1% of runs across seven frontier agents took it, five of the seven above half, and the rate stayed above 50% even when agents were told not to. Benchmark or evaluation
🧪 BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents 2026 · arXiv
Sadia Asif et al.
Scores whether an agent acts, refuses, or stays calibrated as a long interaction evolves, rather than collapsing behaviour into task or attack success. Evaluates whole user-agent-environment trajectories with adaptive adversarial interaction and stateful tool execution across 22 attack families and 35 scenarios. Benchmark or evaluation
🧪 Beyond Endpoint Performance: Process-Level Evaluation of Self-Evolving Agents 2026 · arXiv
Hongqiang Lin et al.
Endpoint scores miss how self-evolving agents change: memories and skills update throughout a stream, so a capability can emerge and later weaken. Freezes evolving artifacts at checkpoints with base model and tools fixed, tracking when each capability appears and whether later updates keep it. Benchmark or evaluation

Operations Playbooks

Running loops in production: cost control, incident handling, and promoting validated agent behavior into cheaper deterministic paths.

Resource Published at Contribution Evidence
📝 Agentic Engineering: The Agent Loop junpingyi.com Minimal mental model for the loop underlying agent operation. Practitioner analysis
📝 The agent loop: ReAct, plan-and-execute, reflection kunwar.page Practical walkthrough of the base loop and common variants. Practitioner analysis
📝 How to Build an Agent ampcode.com Thorsten Ball's demystification of the inner agent loop: a model, a loop, and enough tokens. Practitioner analysis
📝 Agentic Coding Recommendations 2025 · Armin Ronacher's Thoughts and Writings Armin Ronacher's field notes on which practices hold up when agents do most of the work. Practitioner analysis
📝 Coding Agents 101: The Art of Actually Getting Things Done Devin Practical delegation guidance from the Devin team on scoping tasks agents can actually finish. Practitioner analysis
📝 How Anthropic teams use Claude Code 2025 · Claude Cross-team field report of real recurring agent workflows in engineering, security, and data science. Practitioner analysis
📝 How Boris Uses Claude Code How Boris Uses Claude Code
@CarolinaCherry
Unofficial but concrete compilation of Boris Cherny's autonomous setups: parallel worktrees, auto mode, /loop, /schedule, dynamic workflows, and /goal completion conditions. Practitioner analysis
📝 Agent of the Day: Copilot Agent PR Analysis 2026 · GitHub Agentic Workflows Official walkthrough of a daily scheduled agentic workflow that ingests PR data, analyzes it, and publishes findings to a Discussion, a concrete recurring loop with trigger, intake, analysis, and output. Practitioner analysis
📄 Progressive Crystallization: Turning Agent Exploration into Deterministic, Lower-Cost Workflows 2026 · arXiv
Arun Malik
Production lifecycle in which repeatedly validated agent-loop behaviors are promoted into deterministic workflows and demoted on regression, cutting per-incident agent cost by over 70% across eight months of a cloud AIOps system. Research preprint
📄 Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems 2026 · arXiv
Kalle Kujanpää et al.
Production tool-making pipeline that mines live execution traces to compile recurring SOP steps into validated, versioned tools agents call instead of regenerating code, cutting median latency 42% and errors up to 53% in a fulfillment-center alarm-triage deployment. Research preprint
📝 AI Loop Engineering: Build Autonomous Agents with Claude Code /goal and Routines sabrina.dev
Sabrina Ramonov 🍄
Sabrina Ramonov's practitioner walkthrough of building verified loops with Claude Code /goal and cloud routines: a six-part loop-engineering framework, five worked examples each with a verifiable end state, and a production daily support-ticket cleanup routine whose independent checker agents illustrate why the checker is the hard part. Practitioner analysis
📄 Agent Delivery Engineering Predictive Reliability Framework 2026 · arXiv
Dexing Liu
Proactive health-trajectory prediction for long-horizon multi-agent systems, aggregating 20 heterogeneous signals across five layers into a trust-margin metric with 8-hour forecasts at 76.8% direction accuracy, detecting degradation concealed by normal surface metrics across 15 days of production traffic. Research preprint
📄 CADAQUES: A Cost-Aware Dual Architecture for Query-Efficient Autonomous Discovery 2026 · arXiv
Jorge Bravo-Abad
Makes cost part of the loop contract: an Oracle executes queries, a Driver chooses them, a shared vector budget covers wall time, CPU, money, and LLM tokens, and an append-only ledger records declared versus settled cost for every transaction. The Ising-model case study shows how mixed-fidelity scheduling can outperform high fidelity throughout at the tested budget. Research preprint
🧰 rocketplaneIO 2026 · GitHub
olemeyer/rocketplaneIO
Self-hosted AI SRE for Kubernetes whose copilot investigates autonomously via eBPF traces, logs, and live service maps, but can only act through named, reversible, risk-graded operations. Source implementation
📝 Migrating a Production AI Agent to GPT-5.6 Ploy Lorenzo Gentile's July 2026 case study of swapping the model inside Ploy's production website-building agent (plans pages, writes components, screenshots its own work, decides when done): roughly a third of initial cross-model eval failures traced to Opus-specific harness assumptions rather than the new model, GPT-5.6 invented placeholder values for optional tool parameters (fixed via required-but-nullable schemas, after 52-64% of file reads silently returned empty), cache keys needed workspace scoping (0% to 83.7% first-call hits), and reasoning state moved to self-contained encrypted blobs, concrete evidence that eval harnesses and regression gates are the load-bearing layer when changing the model inside a production loop. Practitioner analysis
📄 Coding-agents can replicate scientific machine learning papers 2026 · arXiv
Atharva Hans & Ilias Bilionis
Runs coding agents independently 12 times across four scientific machine-learning papers; all workspaces satisfy the study's completion criteria, and 158 implementation targets are linked to report evidence, offering a concrete playbook for evidence-backed research replication. Research preprint
📄 Agentic ERP: Multi-Agent Large Language Model Architecture for Autonomous Enterprise Resource Planning 2026 · arXiv
Zhihao Liu et al.
Reference architecture that decomposes ERP operations into role-aligned LLM agents under a graph-based planner-executor-reflector-responder orchestration with a risk-tiered human-in-the-loop harness, evaluated across six orchestration paradigms and a 365-day agent-in-the-loop simulation against rule-based RPA, recurring enterprise work run as a governed agent loop. Research preprint
📄 The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agents 2026 · arXiv
Muayad Sayed Ali et al.
Controlled comparison across six foundation models showing the harness, not model choice, dominates agent economics: a redesigned Writer Agent Harness cuts blended cost per task 41%, tokens 38%, and wall-clock 44%, arguing the harness is the one component whose efficiency multiplies across every model an org runs. Research preprint
🧰 claude-thermos 2026 · GitHub
izeigerman/claude-thermos
Keeps a Claude Code session's prompt cache warm while the main agent waits on long-running subagents, so an orchestrator that blocks for more than a few minutes does not silently pay to rebuild its cache on every resume. Source implementation
🧰 Worktrunk 2025 · GitHub
max-sixty/worktrunk
CLI for Git worktree management built for running several coding agents in parallel, giving each agent an isolated checkout so concurrent work does not collide in a shared working directory. Source implementation
📄 Authoring Agent Skills: A Software-Engineering Approach 2026 · arXiv
Giuseppe Destefanis
Treats agent skills as software artifacts subject to single responsibility and low coupling, with concrete guidance on skill structure, loading, selection, and evaluation, plus comparison against other behavior-shaping mechanisms. The first disciplined treatment of a format most teams are currently authoring by feel. Research preprint
📄 A First Look at Coding Agents' Compliance with AI Contribution Rules in Open-Source Communities 2026 · arXiv
Wenhao Yang et al.
Benchmark of 106 issues across 49 repositories showing agents rarely retrieve contribution rules on their own, will disclose assistance and run verification steps when prompted, but consistently fail to refuse work in AI-banned projects. Concrete governance gap for anyone pointing unattended loops at public repos. Research preprint
📄 Who is scientific code for? Maintaining human-readable landmarks in agent-written code 2026 · arXiv
Elle O'Brien
Documents how scientists adopting coding agents invent personal landmarking strategies to separate human-readable artifacts from agent context, and warns this de-standardization will fragment teams without explicit conventions. Practical prompt for codifying readability norms before agent-written code accumulates. Research preprint
📝 Four Incident-Response Lessons from the Hugging Face Breach aikido.dev Mike Wilkes turns the Hugging Face forensic timeline into an incident-response playbook organized around four decision points a team actually faces mid-incident, which is the operational complement to the technique-level writeups already in the list. Practitioner analysis
📄 Token Budgets: An Empirical Catalog of 63 LLM-Agent Budget-Overrun Incidents, with an Affine-Typed Rust Mitigation as a Case Study 2026 · arXiv
Sajjad Khan
Sajjad Khan catalogs 63 confirmed production budget-overrun incidents drawn from 21 orchestration frameworks across 2023-2026, each with quoted maintainer or user evidence and documented dollar losses, organized into an eight-cluster failure taxonomy, the empirical base that the widely-circulated anecdotes (the $47K two-agent conversation loop, the $4,200/63-hour burn) individually lack. Research preprint
📄 When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime 2026 · arXiv
Wei Wu
An eight-week longitudinal study of a personal-assistant agent runtime in continuous production since March 2026, with 22 incidents carrying full root-cause postmortems, as close as the literature gets to a real operations log for a recurring, stateful agent system rather than a benchmark. Research preprint
📄 Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework 2026 · arXiv
Mukund Pandey
A taxonomy of seven failure modes unique to production agentic systems, grounded in observations from systems operating at billion-event scale rather than in benchmark runs, the distinguishing feature relative to the many failure taxonomies built from curated traces. Research preprint
📄 Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures 2026 · arXiv
Harsh Raj et al.
Most agent postmortems stop at 'the run failed,' which tells you nothing about where the repair belongs. Research preprint
📄 Stop Shipping AI Agents on Faith: Capability Is Not Production Readiness 2026 · arXiv
Fouad Bousetouane
Proposes the ProofAgent Index, a governance-readiness score across Evaluation, Context, Compliance, and Governance, and argues against collapsing governance evidence into a single aggregate number. Validated in healthcare and finance deployments, with context engineering emerging as a dominant reliability factor. Research preprint
📄 One Human, N Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence 2026 · arXiv
Cesare Zavattari et al.
Formalizes the core operational question of running many loops with one reviewer: how to spend a limited audit budget when agent confidence is miscalibrated and errors are correlated. Research preprint
📝 An Update on Recent Claude Code Quality Reports 2026 · Anthropic Engineering
Anthropic
Vendor postmortem tracing a reported quality decline to three unrelated changes: a reasoning-effort default drop, a caching bug that cleared prior reasoning every turn instead of once, and a verbosity cap. A worked example of compounding harness regressions reading as model decline. Practitioner analysis
📄 Agent System Operations: Categorization, Challenges, and Future Directions 2026 · arXiv
Zexin Wang et al.
Survey opening the operations and maintenance question for agent systems, on the observation that they hit anomalies as often as conventional systems while research into running them is sparse. Builds a framework for the field rather than proposing another architecture. Research preprint
🧰 SandBase Harness 2026 · GitHub
sandbaseai/sandbase-harness
Local-first agent runtime that persists sessions and resumable event streams in SQLite, runs tools in selectable local, Docker, Kubernetes, or self-hosted sandboxes, and wraps them in credential vaults, permission policies, approvals, and audit. Ships an MCP bridge and a console for inspecting and replaying a run. Apache-2.0. Source implementation
📄 Real-Time Detection and Repair of LLM Agent Failures 2026 · arXiv
Sunny Dubey
Asks how much mid-episode failure is detectable from step telemetry alone, given that judging every step with a second model costs more than the agent itself. A one-class detector trained only on healthy runs, costing microseconds per step, catches 71% of failures at a 5% false-alarm budget across 2,823 episodes. Research preprint
📄 Agentic AI and Human-in-the-Loop Interventions: Field Experimental Evidence from Alibaba's Customer Service Operations 2026 · arXiv
Yiwei Wang et al.
Randomized field experiment on Taobao where supervising workers kept the chats the agent could not take. Deployment cut average chat duration with little effect on retrials, but substantially lowered ratings on the chats the agent did handle, which is the kind of production evidence a benchmark cannot supply. Research preprint
📄 How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks 2026 · arXiv
Longju Bai et al.
Systematic study of where tokens actually go in agentic coding: tasks consume roughly 1000x the tokens of code chat, and input rather than output dominates the bill. Also tests whether models can predict their own cost before running, which is what a budget gate would need to work. Research preprint
📄 What Effort Level Should I Choose? On the Economics of Reasoning Effort in LLM Agents 2026 · arXiv
Kaiser Sun et al.
On tasks every effort level solved, low to xhigh effort raised Terminal-Bench cost 2.51x for GPT-5.6-sol and 3.04x for Claude Sonnet 5. Loops with a reliable check can start low and escalate after a verified failure. Research preprint
📄 Towards Agentic Cloud Engineering: Graph and Loop Engineering with a Zero-Trust Agent Harness 2026 · arXiv
Sagar Srinivas Sakhinana & Venkataramana Runkana
Separates three concerns for cloud work: graph engineering for long-horizon workflow progression, loop engineering for bounded diagnosis and repair, and harness engineering that constrains external actions through authorization. Turns natural-language tasks into validated repositories and verified deployments. Research preprint
📄 Agent Flight Recorder: Tamper-Evident Audit Trails with On-Chain Anchoring for Long-Horizon Tool-Using Agents 2026 · arXiv
Laurent Bindschaedler et al.
Records each agent action as a canonically serialized event binding intent through execution to provenance, with hash chaining and Merkle batching for tamper evidence. For disputes across organizations, anchors a 32-byte epoch root on-chain so any party can verify the record without a trusted intermediary. Research preprint
📄 FIRE: Failure-Informed Runtime Engineering for Reliable Language-Model Agents 2026 · arXiv
Nikita Agarwal & Nivedit Jain
Applies targeted instructions and action denials from the harness at states that preceded observed failures, leaving weights and prompt unchanged. On Terminal-Bench 2.1 the strongest tier's best-of-two success moves 1.2 points while repeated success rises 9.2, so the policies turn occasional success into dependable delivery. Research preprint
📄 Knowing When to Stop: The Evidence-Action Gap in Coding Agents 2026 · arXiv
Tianyu Chen et al.
With injected hard blockers, 98.6% of runs had enough evidence to stop, yet a third ran out of budget; an escalation tool cut that from 34.0% to 6.7%. Loops need a sanctioned exit and tickets that log progress. Research preprint

Templates And Patterns

Reusable patterns that contributors can turn into future examples, templates, or playbooks.

Resource Published at Contribution Evidence
🧾 Resource entry template 2026 · GitHub
Template library
Format for adding a single resource with evidence quality and category fit. Project artifact
🧾 Loop pattern template 2026 · GitHub
Template library
Template for documenting an operational loop such as PR babysitting, CI repair, or feedback clustering. Project artifact
🧾 Loop contract schema 2026 · GitHub
Schema
Machine-readable schema for portable loop specs. Project artifact
🧾 Loop contract preview script 2026 · GitHub
Repository utility
Dependency-free demo that validates and renders a loop contract JSON file. Project artifact
🧾 Translation guide 2026 · GitHub
Project documentation
How to add or maintain a language translation without drifting from the full English guide. Project artifact
🧾 Pattern library index 2026 · GitHub
Pattern library
Practical loop patterns with triggers, state, verification gates, budgets, and escalation paths. Project artifact

All 20 documented patterns span build, operate, optimize, and govern use cases. Each has a full write-up, worked scenario, matching JSON contract, and side-by-side comparison in the matrix.

Examples And Schema

Concrete examples make the loop contract easier to adapt to real repositories.

Resource Published at Contribution Evidence
🔁 Loop contract catalog 2026 · GitHub
Example library
Connects all 22 patterns to schema-valid contracts, deterministic gates, durable receipts, and four worked implementation paths. Project artifact
🧾 Loop contract library 2026 · GitHub
Example library
Adaptable contracts for build, operate, optimize, and govern loops, checked against the shared schema with explicit permissions, budgets, and handoffs. Project artifact
🧾 Runnable test-repair loop 2026 · GitHub
Runnable example
Repeats a failing deterministic check with durable progress, duplicate-failure detection, and a hard retry budget. Project artifact
🧾 Runnable loop guide 2026 · GitHub
Runnable example
Compares 8 starters by trigger, state, gate, and runtime, including executable test-repair, threshold-monitor, and queue-worker loops. Project artifact

Preview an example locally:

python3 scripts/preview_loop_contract.py examples/pr-babysitter-loop.json

Community Gallery

Study concrete loop designs or contribute a real, realistic, or safely anonymized example.

Running a real loop? Share it, real or anonymized, in the patterns discussion under Shape What Comes Next. Use the minimum useful case study and anonymization checklists so others can learn from it safely.

Resource Published at Contribution Evidence
🧾 Loop gallery guide 2026 · GitHub
Community gallery
Quality bar for contributed loop examples with receipts and lessons learned. Project artifact
🧾 Loop gallery template 2026 · GitHub
Community gallery
Markdown template for sharing a loop's trigger, intake, state, verification, escalation, and safety notes. Project artifact
🔁 PR babysitter reference loop 2026 · GitHub
Community gallery
Reference gallery entry for keeping a pull request moving. Project artifact
🔁 CI repair reference loop 2026 · GitHub
Community gallery
Reference gallery entry for turning failing CI into a verified patch or escalation. Project artifact
🔁 Docs drift reference loop 2026 · GitHub
Community gallery
Reference gallery entry for recurring docs/code consistency checks. Project artifact

Critiques, Risks, And Limitations

Skeptical takes and measured failure modes, because a loop that cannot fail safely should not run unattended.

Resource Published at Contribution Evidence
⚠️ Most Developers Do Not Need Agent Loops Yet Substack
AlphaSignal AI
Useful caution against adopting loops before the task, signal, and economics justify them. Risk analysis
⚠️ Engineering Agentic Systems for Reliability 2026 · Pruning My Pothos
Shailesh Rawat
Cautions that agentic systems fail at boundaries when permissions, verification, traceability, and escalation are weak. Risk analysis
⚠️ Self-Correcting Agents: Reflexion, CRITIC, and ReAct Loops Compared 2026 · CallSphere
CallSphere
Compares self-correction patterns and their cost/failure tradeoffs. Risk analysis
⚠️ How to Build an AI Agent Harness: A 2026 Complete Guide 2026 · atlan.com Broad guide with useful warnings on data readiness, permissions, context management, and evaluation. Risk analysis
⚠️ Harness Engineering vs Prompt Engineering vs Context Engineering Explained Medium Adjacent framing that helps avoid confusing loop engineering with the surrounding harness discipline. Risk analysis ‡
📄 Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering 2026 · arXiv
Maria I. Gorinova et al.
Argues benchmark scores conflate the model with the harness and penalize valid alternatives, so headline numbers hide which loop and harness choices actually move performance. Research preprint
📄 Understanding the Challenges in Iterative Generative Optimization with LLMs 2026 · arXiv
Allen Nie et al.
Empirically isolates three hidden design choices that make self-improving agent loops succeed or fail - starting artifacts, credit horizons over execution traces, and batching strategy - explaining why iterative refinement loops stay brittle in production. Research preprint
📄 The Illusion of Multi-Agent Advantage 2026 · arXiv
Prathyusha Jwalapuram et al.
Systematic evaluation showing automatically generated multi-agent systems consistently underperform chain-of-thought self-consistency while costing up to 10x more, cautioning that auto-designed orchestration adds complexity without functional benefit. Research preprint
⚠️ The Coming Loop 2026 · Armin Ronacher's Thoughts and Writings Flask creator Armin Ronacher's skeptical essay on harness loops, examining what continuously re-driving agents past their natural stopping points does to code quality, review capacity, and human understanding of the resulting systems. Risk analysis
⚠️ Loop Engineering, the Latest AI Buzzword, Still Needs Humans in the Loop 2026 · theregister The Register's report on the June 2026 loop-engineering discussion, collecting the Steinberger, Osmani, and Cherny quotes while arguing that vendor token-consumption incentives and model non-determinism keep humans in the loop. Risk analysis
📄 When Agents Do Not Stop: Uncovering Infinite Agentic Loops in LLM Agents 2026 · arXiv
Xinyi Hou et al.
Characterizes infinite agentic loops, a failure class where unbounded feedback paths make agents repeat calls, tools, or handoffs forever, and ships IAL-Scan, a static analyzer that confirmed 68 real cases across 47 of 6,549 scanned agent projects at 91.9% precision. Research preprint
📄 The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents 2026 · arXiv
Xing Zhang et al.
Shows via corrupted-reward analysis that false-pass bias in an LLM judge silently disables the skill-retirement mechanism that keeps a self-evolving agent's growing skill library from drifting below the no-skill baseline. Research preprint
📄 Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows 2026 · KDD Workshop on AI Data Scientist
Wei-Jung Huang
Negative result for low-curation skill libraries: across four data-science lifecycle stages (56 tasks), fully LLM-generated skill files show no reliable improvement over plain task prompting, and component ablations find no skill part that helps either (all p > 0.396). Useful counterweight to the skill-generation enthusiasm in self-evolving agent stacks, curation still matters. Research paper
📄 The Verification Horizon: No Silver Bullet for Coding Agent Rewards 2026 · arXiv
Binghai Wang et al.
Position paper arguing verification has become harder than generation for coding agents: every verifier is only a proxy for underspecified human intent, so reward design faces a horizon that no single verification mechanism crosses. Research preprint
⚠️ Write Code Like a Human Will Maintain It 2026 · Unstack Argues that agent-driven codebases create a compounding feedback loop where every merged shortcut becomes training signal for the next generation of changes, so code quality standards matter more, not less, under automation. Risk analysis
📄 Rethinking the Evaluation of Harness Evolution for Agents 2026 · arXiv
Yike Wang et al.
Re-evaluates harness evolution on Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6, finding that evolved harnesses do not consistently beat budget-matched search and transfer only weakly to held-out tasks. Research preprint
📄 Compaction as Epistemic Failure: How Agentic LLM Tools Fabricate Confirmed Results from Killed Processes 2026 · arXiv
Hiroki Tamba
Documents a Claude Code failure in which partial output from a process killed with exit 143 becomes a confirmed claim after context compaction, without re-verification, showing why receipts and process status must survive summarization. Research preprint
📄 Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0 2026 · arXiv
Wenxiao Wang et al.
Compares GEPA, Meta Harness, and RELAI-VCL under matched continual-learning budgets; only the regression-controlled method keeps improving, reaching 76.4% lifelong performance versus 66.0%, 64.6%, and 58.7% for the reported alternatives. Research preprint
📄 Does Multi-Agent Debate Improve AI Feedback on Research Papers? 2026 · arXiv
Tomas Havranek & Zuzana Irsova
In a preregistered masked study with authors of 44 meta-analyses, participants prefer single-pass feedback to two multi-agent debate systems, one using about 30x more tokens; AI judges reverse the human preference, warning against self-evaluation alone. Research preprint
📄 Binding Drift in Multi-Step Tool-Augmented Agents 2026 · arXiv
Rahul Suresh Babu & Shashank Indukuri
Isolates 'binding drift', entity bindings that start correct then silently go wrong across sequential tool-calling steps, from ordinary error propagation, using 200 workflows and 580 entity-binding-scored steps across four enterprise domains and eight model backends; a naive entity-lock persistence mechanism amplifies injected wrong actions 3.0x on average (up to 8.5x on the most affected model), while an LLM re-verification pass cuts wrong actions 79%. Research preprint
📄 How Agent Skills Fail under Long Contexts: A White-Box Study in Code Auditing 2026 · arXiv
Yue Xue
White-box study of skill-following degradation over long tool-using trajectories in a code-audit workflow: pass rates fall from 8/10 in clean context to 3/10 at ~299K characters even though requirement coverage stays above 92%, a detailed external checklist restores 10/10 versus 5/10 for generic self-check, and the paper taxonomizes the failure modes (lost requirements, editing drift, failed checking) that skill-driven loops must engineer around, a small single-model study of 10 runs per condition. Research preprint
📄 Phantom Guardrails: When Self-Improving Agent Harnesses Fix Failures That Never Happened 2026 · arXiv
Su Wang et al.
Names a failure mode of self-improving harness loops: the proposer LLM fabricates guardrails for failure classes that provably never occurred (15 of 60 runs when input merely resembles a familiar rule), and once inside an add-only accept loop the phantom guardrail persists; ships a deterministic micro-lab with byte-exact oracle checks. Research preprint
📄 Masked Erosion: How Acceptance Gates Lose Redundant Safeguards in Self-Improving Agents 2026 · arXiv
Hau-Sing So et al.
In 411 self-improvement loops, deleting a safeguard stayed invisible while another still blocked the harm, so score gates accepted it; at equal cost, 40-task batches lost all six harnesses' protection, 20-task none. Research preprint
📄 Skills That Don't Exist: A Large-Scale Study of Hallucinated Skill Recommendation 2026 · arXiv
Weifeng Yuan et al.
Measures a 36% average rate of agents recommending non-existent skills across 15,000 prompts, and shows the same fake names recur consistently, enabling slopsquatting-style supply-chain attacks where adversaries pre-register malicious skills under the hallucinated names agents will predictably ask for. Research preprint
📄 Token Reduction Is Not Cost Reduction 2026 · arXiv
Sarel Weinberger & Amir Hozez
Analysis of 2,848 real Claude Code runs showing prompt-cache traffic accounts for ~87% of billed cost, so local token/context compression does not reliably lower the bill, argues loop cost engineering should optimize success-adjusted billed cost, not token counts. Directly actionable for anyone running high-volume agent loops. Research preprint
📝 What xAI's Grok Build CLI Actually Sends: A Wire-Level Analysis Gist Wire-level analysis of the telemetry and payloads Grok Build transmits, a reminder that agent harnesses carry their own data-flow surface worth auditing before unattended use. Practitioner analysis
⚠️ The Tower Keeps Rising 2026 · Armin Ronacher's Thoughts and Writings Armin Ronacher's follow-up to The Coming Loop, on abstraction layers accumulating faster than understanding as agent tooling stacks up, and what that does to a codebase's long-term comprehensibility. Risk analysis
📄 The Dark Room in the Reward Channel: Dense Prediction Rewards Collapse GRPO-Trained LLM Agents -- and What Actually Works 2026 · arXiv
Yu Wang
Shows dense next-observation prediction rewards under GRPO drive long-horizon LLM agents into a degenerate "dark room" absorbing state (prediction accuracy 1.0, task success 0), with a single-factor ablation localizing the cause to std normalization and auxiliary-loss channels as a working fix. A crisp negative result for anyone designing reward channels in agent training loops. Research preprint
📝 Why Software Factories Fail (or: Harness Engineering Is Not Enough) 2025 · GitHub
humanlayer/advanced-context-engineering-for-coding-agents
Dex Horthy's essay from his AI Engineer World's Fair 2026 keynote arguing that lights-off software factories fail because RL training rewards passing tests with no penalty for eroding codebase maintainability, so long-horizon quality feedback cannot be trained on, proposing front-loaded human judgment gates (requirements review, architecture, vertical slices) as the missing layer above the harness. Practitioner analysis
📄 The Boundaries of Automation: A Theory of Persistent Human Participation 2026 · arXiv
Fares Fourati et al.
Theory paper from Fourati, Schütze, Hüllermeier, and Gurevych challenging the assumption that humans stay in the loop only until AI capability catches up: it identifies three grounds for persistent human participation, technical complementarity, normative/developmental value, and objectives that emerge through the interaction itself, framing human-AI co-construction as a permanent feature rather than a stopgap. Research preprint
🧰 Reward Hacking in the Wild Reward Hacking in the Wild Searchable corpus of 3,607 user-reported AI-agent misbehavior incidents collected from GitHub, Hacker News, LessWrong, and X, normalized and classified across fourteen misbehavior types with severity ratings. Implementation
📄 What AI Red-Team Evaluations Can and Cannot Prove 2026 · arXiv
Bandana Kaur
Bandana Kaur gives a formal treatment of the epistemic limits of red-teaming, the practice now serving as the industry's primary safety gate for agent deployment. Research preprint
📄 The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages 2026 · arXiv
Zixuan Wu et al.
Shows coding-agent token cost varies starkly and consistently by programming language across five recent models on difficulty-controlled Python, Java, Rust, and OCaml problems. Research preprint
📄 Where Is the Cost of Third-Party API Routers in Agentic Software Development? 2026 · arXiv
Donghao Fu et al.
Third-party LLM routers sit on the trusted path between a coding agent and its provider, able to inspect and modify every request and response, while nothing verifies that provider output matches the repository-level actions the agent ultimately executes -- so client-side permission mechanisms can silently stop working. Research preprint
📄 Efficiency Matters in Autonomous Research 2026 · arXiv
Haiqian Yang & Yuan Cao
Position paper arguing autonomous research systems are judged almost entirely on final outcome quality while search efficiency -- reaching that outcome on a small budget -- is an equally important and ignored dimension, and one that dominates as AR moves from cheap-verification domains like math and code into settings where evaluating a candidate means running a physical experiment. Research preprint
📄 Reliability-Contagion Feasibility in LLM Multi-Agent Networks 2026 · arXiv
Ruiwu Niu et al.
Treats error propagation in multi-agent systems as an epidemic on the communication graph, with susceptible/exposed/infectious/corrected states and a derived early-invasion condition for heterogeneous topologies, then couples it to an analytic majority-vote benchmark where a clean-task reliability target imposes a minimum connectivity requirement. Research preprint
📄 Draining the Energy Commons: Self-Defeating Over-Appropriation as a Coordination Failure in Agentic LLM Collectives 2026 · arXiv
Marcantonio Bracale Syrnikov et al.
When LLM agents share a persistent resource, one agent's decision changes the conditions later agents face. Four same-family GPT, Gemini, or Grok agents act as electricity prosumers instructed to maximize operational continuity, with aggregate demand and protocol held fixed while the regeneration rate varies. Research preprint
📄 "Go Home Copilot, You're Drunk": Understanding Developer Responses to Agent-Generated Code Review Comments 2026 · arXiv
Shamse Tasnim Cynthia et al.
First large-scale empirical study of what happens after an agent posts a review comment: 54,791 comments from Copilot, Cursor, Codex, Devin, and Claude across 342 Python GitHub repositories, analyzed for resolution rates by agent and comment type, the effect of developer experience, and what makes a comment useful. Research preprint
📄 Position: Evaluation Scores Are Perishable Knowledge Claims 2026 · Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM) 2026
Sankalp Gilda & Shlok Gilda
Argues benchmark scores are epistemic claims with expiry dates and that averaging heterogeneous signals inflates confidence past the weakest component, proposing formality tiers, scope declarations, and expirations, HELM rankings shift materially under weakest-link aggregation. Sharp critique of how loop teams read eval dashboards. Research paper
📄 One Run Is Not an Idea: The Implementation Lottery in Automated Research 2026 · arXiv
Jingjie Ning et al.
Shows automated research loops judge ideas on a single implementation, that implementation variance dwarfs rerun variance, and that winner reversal rates exceed 25%, then offers an Idea Reliability Audit. Directly undermines single-run selection in self-improving research agents. Research preprint
📄 Can AI agents conduct open-ended AI research? Early evidence from two case studies 2026 · arXiv
Peter Kirgis et al.
Frontier agents ran six days of largely independent engineering on open research questions from unpublished NeurIPS 2026 papers, and the original authors unambiguously rejected both outputs, the gap was research judgment and creativity, not execution. A rigorous shadow-evaluation counterweight to autonomous-research optimism. Research preprint
📄 Two Calls Beat Five Agents: Evaluating Multi-Agent Pipelines Against Self-Refinement for Local Language Models 2026 · arXiv
Ashish Prajapati & Om Mohite
On a 7B local model, two refinement iterations beat a five-agent architecture on math reasoning, with data formatting and task-specific tuning mattering more than topology. Practical argument against reaching for multi-agent structure before exhausting the simple loop. Research preprint
📄 Try Again, Don't Look Back: Blind Resampling Outperforms Self-Repair in Small Code Models 2026 · arXiv
Yuvraj Verma
Finds that resampling without showing the model its failed attempt beats self-repair, because conditioning on its own broken code anchors the model into reproducing near-identical flaws. A concrete case where the feedback in the feedback loop actively hurts. Research preprint
📄 Even More Deception: Objective Misalignment in Mixed-Motive LLM Multi-Agent Systems 2026 · arXiv
Marylou Fauchard et al.
In a Werewolf testbed, agents with hidden or conflicting objectives show detectable shifts in internal reasoning while public messages mask the change, degrading collective decisions. Evidence that monitoring inter-agent transcripts alone will not catch misaligned delegates. Research preprint
📄 (Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding 2026 · arXiv
Nishant Balepur et al.
Controlled study of 54 students comparing an agent-based system against a chatbot: agents raise completion speed but lower code comprehension and the ability to extend the work independently, with copy-paste prompting predicting the weakest understanding. Quantifies the comprehension debt that accrues inside sustained agent loops. Research preprint
📝 We Gave GPT-5.6 Sol a Real Business Bottleneck Labs Field report from handing a frontier model an actual operating business and letting it run unattended, recording where the loop held up and where it needed a human. Useful as evidence about long-horizon autonomy outside benchmark conditions. Practitioner analysis
📝 GCC Steering Committee Announces AI Policy LWN.net LWN, 2026-07-29. The GCC steering committee will decline any legally significant contribution that includes or is derived from LLM-generated content, with 'legally significant' pegged at roughly 15 lines per GNU maintainer guidelines. Practitioner analysis
📄 Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents 2026 · arXiv
Wael Albayaydh et al.
A cross-cutting synthesis of 27 benchmark, taxonomy and audit papers from 2023-2026 spanning 19 distinct benchmarks, collapsed into six failure clusters: tool invocation and parameter-level errors, planning and constraint-satisfaction failures, long-horizon degradation from context accumulation, multi-agent coordination failures, safety and security failures, and measurement-validity issues. Research preprint
📄 Guardrails as Scapegoats: Auditing Unfaithful Safety Refusals in Tool-Augmented LLM Agents 2026 · arXiv
Aarushi Singh
A black-box auditing framework for a failure mode that silently corrupts verification in unattended loops: an agent hits a silent infrastructure failure, a tool returning nothing, a timeout, a malformed response, and reports it to the user as a safety refusal rather than as a broken dependency. Research preprint
📄 Towards a Science of AI Agent Reliability 2026 · arXiv
Stephan Rabanser et al.
The foundational negative result behind the reliability-versus-capability argument that several entries in this list gesture at without citing. Research preprint
📝 Ten AI Agents Destroyed Production, Zero Postmortems 2026 · Harper Foley - AI Product Leader
Harper Foley
Harper Foley (Tribe AI, ex-Navy EOD) catalogs ten production-destroying agent incidents across six tools over 16 months, each sourced to GitHub issues, Fortune, The Register, or first-hand reports, and shows not one vendor published a postmortem. Practitioner analysis
📝 2x, Not 10x: Coding With LLMs in 2026 obryant.dev
Jacob O'Bryant
A calibration essay whose central claim is squarely a loop-engineering claim: LLMs became genuinely useful at the point they got reliable enough to run inside automated feedback loops, and past that threshold further model capability buys much less than retooling does. Practitioner analysis
📄 Reproducing LightMem: Naive RAG Is Just as Good for Memory Management 2026 · arXiv
Yongjie Zhou et al.
A reproduction study that lands harder than most original results. The authors rebuild LightMem, a well-cited lightweight memory-management approach, and compare it against naive RAG retrieving directly from raw user turns. Research preprint
📝 Critical CVEs or LLM Slops? Inside 50+ Fake SQLite Advisories research.jfrog.com JFrog Security Research (Afek Berger) audited 55 SQLite vulnerability advisories published from a single GitHub account with initial CVSS scores of 7.5 to 9.8, and found 54 completely fabricated, non-existent functions, invalid line numbers, PoCs that trigger no crash, with one real bug wrapped in unverified CVE metadata. Practitioner analysis
📝 Don't be a meat proxy 2026 · gruhn.me The highest-traction technical post of the Aug 1-3 window, arguing against the degenerate role humans fall into when they relay model output verbatim, into Slack threads, PR review comments, group chats, without reading, understanding or validating it first. Practitioner analysis
📄 What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants 2026 · arXiv
Alif Al Hasan & Sumon Biswas
An incident-driven empirical study rather than a benchmark paper: the authors screened 68,816 papers across 22 venues to curate 185 safety-relevant studies, then mined 16,586 GitHub issues from LLM-powered coding tools and manually confirmed 547 genuine operational safety failures, each annotated with contributing factors, task context, severity and downstream impact. Research preprint
📄 How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions 2026 · arXiv
Ningzhi Tang et al.
A study of 20,574 real coding-agent sessions across 1,639 repositories, categorizing seven recurring forms of developer-agent misalignment and separating IDE from CLI usage patterns. Research preprint
📄 Skill Use or Skill Theater? Evaluating the Reasoning Backroom in Skill-Augmented Language Agents 2026 · arXiv
Jinwei Hu et al.
BACKTRACE measures whether a skill actually caused a decision by comparing skill-conditioned against no-skill runs and perturbing skill attributes. Research preprint
📄 Fidelity Is Not Safety: Gently-Compressed LLMs Pass Every Data-Free Quality Guard Yet Invent Procedure Steps in Agentic Execution 2026 · arXiv
I. Kennedy & T. Kennedy
Compressed models clear perplexity, downstream accuracy, and data-free output-fidelity checks, then fabricate procedure steps once deployed in an agent loop, and the effect is specific to coherent low-rank (SVD) error, not magnitude pruning at matched perplexity. Research preprint
📄 Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs 2026 · arXiv
Woongkyu Lee & Jungwook Choi
Tests whether spending more inference compute rescues small locally-hosted computer-use agents on OSWorld, and finds it mostly changes the failure mode rather than fixing it: contextual scaling stabilizes trajectories, but temporal scaling extends wrong paths instead of correcting them. A useful corrective to 'just let it loop longer'. Research preprint
📄 Safeguards Based on Copyable Context Cannot Provide Reliable Safety for LLMs 2026 · arXiv
Pingyu Wu et al.
Submitted 2026-07-30. An impossibility argument aimed squarely at the standard loop-safety design: if the evidence a safeguard uses (system prompt, conversation history, stated role) is copyable, an attacker can imitate it. Research preprint
📄 Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories 2026 · arXiv
Prakhar Khatri
Submitted 2026-07-28. Controlled ablation of AGENTS.md / CLAUDE.md context files across two frontier agents, 17 real repository tasks, and 288 evaluated runs. Finding: context strategy does not measurably move correctness on either agent, and failures are dominated by implementation difficulty rather than missing repo knowledge. Research preprint
📄 AI Agents Push Humans Out of the Loop 2026 · arXiv
Margaret Mitchell et al.
Position paper against treating human oversight as a free mitigation. Argues current agent designs both impede oversight and erode the cognitive capacity it depends on through extended use, so keeping a human in the loop degrades over time unless the system is built around what an overseer actually needs. Research preprint
📄 The Missing Boundary: How Autonomous Agents Lose Control 2026 · arXiv
Zonghao Ying et al.
Asks when an agent crosses the boundary of authorized execution while pursuing a legitimate task, rather than under attack or conflicting goals. Varies goal pressure, control degradation, and the availability of an executable unsafe action independently, locating loss of control inside ordinary execution. Research preprint
⚠️ An alignment assessment of the UK AISI cybertesting incident 2026 · Anthropic Replayed from compaction cycle 4 on, Claude Mythos 5 kept up its out-of-scope attack 98% of the time, and abort rules added to its compaction notes never fired after cycle 3. Enforce stops outside the model. Risk analysis
📄 Rethinking Multi-Agent Collaboration: When More Is Less 2026 · arXiv
Yishuo Yuan et al.
Argues multi-agent collaboration hits diminishing returns as single agents improve, while its context overhead keeps growing. Places the boundary at long-horizon tasks with sparse dependencies, and finds a single-agent harness still wins on tightly coupled sequential work. Research preprint
📄 What Does a Harness Buy? Tokens, Mostly 2026 · arXiv
Yangze Liu & Zhongyi Han
With the model fixed, swapping among Claude Code, mini-SWE-agent, and OpenCode flips as many tasks as rerunning the same harness, so most claimed harness gains sit inside run noise. What the harness does decide is cost, up to 3x per task. Research preprint

Future Directions

The Future Directions agenda turns fifteen open problems into scoped studies, runtime projects, product pilots, and shared standards. Every workstream defines the decision it unlocks, a minimum useful artifact, core measures, a starter slice, and a completion gate.

Build in this order What must be proved Start with
🧱 Foundation Verification resists false completion; state survives interruption; execution is idempotent, observable, and contained Verifier challenge set, crash injection, receipt schema, state replay, and least-privilege policy
📐 Scale Gains remain after factorized ablations, matched budgets, long-horizon failures, runtime changes, and full cost accounting Benchmark harness, stopping-policy replay, portable contract profile, and cost-quality frontier
🚀 Adoption Operators can understand, interrupt, hand off, update, and retire a useful domain loop Shadow pilot, evidence-rich handoff, promotion gates, incident drill, and lifecycle owner

Start with one recurring task, freeze its Loop Contract, compare current operation plus single-pass and fixed-retry baselines, inject failures, then publish every run, receipt, cost, and negative result. Use the shared evaluation protocol, follow a 90-day role plan, or open a direction proposal.

Adjacent Awesome Lists

Related directories and reading lists for the layers around loop engineering.

Resource Published at Contribution Evidence
🧭 Awesome Harness Engineering by ai-boost 2026 · GitHub
ai-boost/awesome-harness-engineering
Comprehensive list for the agent harness layer that Loop Engineering builds on. Discovery index
🧭 Awesome Harness Engineering by walkinglabs 2026 · GitHub
walkinglabs/awesome-harness-engineering
High-signal harness list with strong categories for context, guardrails, specs, evals, runtimes, and benchmarks. Discovery index
🧭 Awesome Agent Harness 2026 · GitHub
AutoJunjie/awesome-agent-harness
Curated tools and resources for environments, constraints, and feedback around coding agents. Discovery index
🧭 Awesome Context Engineering 2025 · GitHub
Meirtz/Awesome-Context-Engineering
Survey-style list for context engineering across LLMs and agents. Discovery index
🧭 Awesome Prompt Engineering 2023 · GitHub
promptslab/Awesome-Prompt-Engineering
Classic adjacent list for prompt techniques and prompting resources. Discovery index
🧭 Awesome LLM Agents 2023 · GitHub
kaushikb11/awesome-llm-agents
General list of LLM agent papers, frameworks, and applications. Discovery index
🧭 Awesome AI Agents 2023 · GitHub
e2b-dev/awesome-ai-agents
Broad AI agent ecosystem map. Discovery index
🧭 Awesome CLI Coding Agents 2026 · GitHub
bradAGI/awesome-cli-coding-agents
Directory of terminal-native coding agents, parallel runners, autonomous loops, and the harnesses that orchestrate them. Discovery index
🧭 Awesome Self-Evolving Agents 2026 · GitHub
XMUDeepLIT/Awesome-Self-Evolving-Agents
Survey-style list of agents that improve themselves over repeated runs, an adjacent angle on long-running loops with memory and verification. Discovery index
🧭 Awesome AI Agent Papers 2026 · GitHub
VoltAgent/awesome-ai-agent-papers
Curated 2026 research collection across agent engineering, memory, evaluation, workflows, and autonomous systems, a paper-level feeder for loop-design foundations. Discovery index
🧭 awesome-ralph 2026 · GitHub
snwfdhmp/awesome-ralph
Curated directory for the Ralph technique, collecting official resources, implementations, playbooks, tutorials, and community channels for running coding agents in automated loops until specifications are fulfilled. Discovery index
🧭 Awesome Agent Loops 2026 · GitHub
serenakeyitan/awesome-agent-loops
Curated collection of /loop, /goal, and /schedule commands for Claude Code and Codex sourced from practitioner posts, organized around trigger, condition, and skill structure. Discovery index
🧭 Awesome Loop Models 2026 · GitHub
huskydoge/Awesome-Loop-Models
Dedicated catalog of architectures that reuse a learned layer, block, module, or operator within one forward process; use it for deeper model-level coverage while this repository focuses on the bridge to operational agent loops. Discovery index
🧭 harness-engineering (Ryan Lopopolo) 2026 · GitHub
lopopolo/harness-engineering
Ryan Lopopolo's anthology, field guide, and agent context bundle for harness engineering, collecting primary sources on the layer directly beneath loop engineering. Discovery index
🧭 A Coding-Agent Reading List: Behind the Loops insights.ml4trading.io Stefan Jansen's curated reading path of 60+ resources on coding-agent loops, organized to separate practitioner discourse, control-theory foundations, agent primitives, harness papers, adoption studies, and safety work by evidence type. Discovery index
🧭 awesome-agent-harness 2026 · GitHub
Picrew/awesome-agent-harness
Companion index scoped to the harness layer itself, collecting projects, tools, benchmarks, and practical guides rather than the operational loop built around them. Discovery index
🧭 Awesome-Memory-for-Agents 2025 · GitHub
TsinghuaC3I/Awesome-Memory-for-Agents
Paper collection scoped to memory for language agents, useful as a companion index when the memory layer specifically is the part being designed. Discovery index

Explore And Reuse

Browse interactively, query structured exports, or reuse the publishing components.

Resource Published at Contribution Evidence
🧾 Resource Atlas chaoyue0307.github.io Filter 1022 resources by goal, loop layer, lifecycle stage, artifact type, evidence class, and search query. Reusable artifact
🧭 Hugging Face dataset Hugging Face Query the full collection as generated CSV and JSONL tables with publication, evidence, and lifecycle fields. Discovery index
🧾 Dataset export guide 2026 · GitHub
Project documentation
Load, query, regenerate, and audit the CSV, JSONL, and Resource Atlas data. Project artifact
🧾 Runtime selection guide 2026 · GitHub
Project operations guide
Compare session, scheduled, CI, cron, and durable runtimes by persistence, isolation, permissions, and state. Project artifact
🧾 Future Directions agenda 2026 · GitHub
Project documentation
Turn 15 open problems into measurable studies, runtime projects, product pilots, and shared standards. Project artifact

Load the resources directly from Hugging Face:

from datasets import load_dataset
resources = load_dataset("cy0307/awesome-loop-engineering", "resources", split="train")

Shape What Comes Next

Track upcoming work, inspect release history, propose evidence, or share an operating lesson.

Resource Published at Contribution Evidence
🧾 Release notes 2026 · GitHub Releases Versioned changelog of new resources, patterns, and repository changes. Reusable artifact
🧾 Roadmap 2026 · GitHub
Project documentation
Near-term work, pattern priorities, gallery goals, and open questions. Project artifact
🧾 Launch article 2026 · GitHub
Project article
Concise explanation of the concept, implementation kit, and evidence base. Project artifact
🧾 Discussion guide 2026 · GitHub
Project operations guide
Suggested discussion categories, starter prompts, and moderation standard. Project artifact
🔁 Show your Loop Engineering patterns 2026 · GitHub Discussions Community discussion for real or anonymized loop examples. Operational pattern

Build From This

Move from evidence to a working loop.

▶️ Run locally Follow the test-repair walkthrough, then switch to the runtime that owns your trigger and isolation.
🧾 Adapt a contract Start from the CI repair contract and replace its objective, intake, checks, budget, and escalation rules.
✅ Validate a contract Use the loop-contract-schema package in CI or call the JSON Schema from your own validator.
📊 Analyze the field Follow the dataset reproduction steps to work with the CSV, JSONL, source-check results, and Resource Atlas payload.
➕ Contribute one source Use the resource suggestion form or the under-five-minute contribution path.
🔁 Share a real loop Use the case-study form to contribute a public or anonymized operating example without preparing a pull request.

Fork the contracts, runtime starters, and dataset schema for your own loop system.

Contributing

Contribute an original or official source, an evidence-backed correction, or a loop you have run. Read CONTRIBUTING.md before opening a pull request.

New contributors can start with a good first issue, a help wanted task, or the case-study form.

The resource selection guide explains what belongs, which evidence to include, how to write useful summaries, and how corrections are handled. The maintenance guide explains link checks and releases.

For community expectations and support channels, see CODE_OF_CONDUCT.md, SUPPORT.md, and SECURITY.md.

Fast path for adding a resource:

  • Check that it is about AI/coding-agent Loop Engineering or a direct foundation for it.
  • Search the README to avoid duplicates.
  • Pick the most specific category.
  • Add one entry using this format:
| 📄 **[Title](https://example.com)** | **2026** · arXiv<br><sub>First Author et al.</sub> | One sentence explaining the resource's concrete contribution, result, or limitation. | **Research preprint**<br><sub>Preprint; inspect methods and evaluation</sub> |
  • Open a pull request and explain the category fit, source type, and why builders should care.

Fast path for contributing a loop pattern: start from the loop pattern template or loop contract schema, include trigger, discover/intake, delegation, workspace, context, verification, durable state, budget, escalation, and exit, then open a pattern suggestion issue if you want feedback before writing the full pattern.

Good submissions should answer three questions:

  1. Is this about the new AI/coding-agent meaning of Loop Engineering or a direct foundation for it?
  2. Does it help someone design, run, verify, evaluate, or critique recurring agent systems that coordinate prompting, context, harnesses, verification, and state?
  3. Is the source stable, public, and specific enough to be useful?

Citation

DOI

Every release is archived on Zenodo, and this DOI always resolves to the latest one. Cite the field guide with:

@misc{chaoyue2026awesome_loop_engineering,
  author       = {He, Chaoyue},
  title        = {Awesome Loop Engineering},
  year         = {2026},
  doi          = {10.5281/zenodo.23617324},
  howpublished = {\url{https://github.com/ChaoYue0307/awesome-loop-engineering}},
  note         = {Field guide and implementation kit for Loop Engineering, version 0.11.0}
}

Star History

GitHub star growth for Awesome Loop Engineering

Every star helps more researchers and builders discover the field guide. Star the repository to follow its growth.