js-lee-AI/awesome-llm-agent-papers
js-lee-AI/awesome-llm-agent-papers has 114 GitHub stars and 41 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-15 15:55 UTC. Last push 2026-11-14.
What you can do
- 📚 A superset of the survey - The 228 references behind “LLM Agents: A Survey,” plus papers that came out after the paper was written. New papers are still being added.
- 🧭 Organized by function - 10 sections following the survey's structure: Surveys, Architectures, Planning, Memory, Tool Use, Multi-Agent, Environments, Applications, Evaluation, Safety.
- ✍️ Annotated - Each entry has a one-line note on what it contributes, its venue and year, and a [code] link where an official implementation exists.
- ⭐ Starter Kit - A 10-paper list for getting oriented, with a note on why each one is worth reading first.
Measured star growth
- in 24h
- 0
- in 7 days
- Update delayed
- in 15 days
- Update delayed
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
LLM agents: curated papers on planning, memory, tool use, multi-agent systems, evaluation, and safety.
AIREADME.md
Original🤖 Awesome LLM Agent Papers
English · 한국어 · 简体中文 · 日本語 · Português · Español
200+ must-read papers, updated regularly: the annotated reading list for building
LLM agents that plan, remember, use tools, and cooperate. Companion to the survey “LLM Agents: A Survey.”
📄 Read the survey → “LLM Agents: A Survey” · PDF in this repo
📖 Online book on GitBook · ⭐ Start with the Starter Kit
LLM agents · LLM agent papers · autonomous agents · agentic AI · multi-agent systems · tool use · ReAct · planning · memory · agent benchmarks · agent safety & prompt injection

✨ Highlights
| What's here | |
|---|---|
| 📚 A superset of the survey | The 228 references behind “LLM Agents: A Survey,” plus papers that came out after the paper was written. New papers are still being added. |
| 🧭 Organized by function | 10 sections following the survey's structure: Surveys, Architectures, Planning, Memory, Tool Use, Multi-Agent, Environments, Applications, Evaluation, Safety. |
| ✍️ Annotated | Each entry has a one-line note on what it contributes, its venue and year, and a [code] link where an official implementation exists. |
| ⭐ Starter Kit | A 10-paper list for getting oriented, with a note on why each one is worth reading first. |
| 🔎 Navigable | A Contents index with per-section counts and collapsible sections. |
Topics covered: cognitive architectures · ReAct & reasoning-and-acting · long-horizon planning · agent memory · tool-augmented LLMs · multi-agent collaboration · web / code / embodied agents · agent benchmarks & evaluation · safety, alignment & indirect prompt injection.
📖 The survey as an online book: LLM Agents: A Survey on GitBook rewrites the paper for readers new to the area, in ten chapters that start from the basic concepts. As in the paper, approaches that fell short get as much space as the ones that worked.
🔁 The list for the follow-up survey: Awesome Agent Loop Papers collects work on the agent loop itself, with 524 papers and 60 open-source projects (frameworks, coding harnesses, memory and sandbox infrastructure, skill libraries, registries). It accompanies The Agent Loop: A Survey of Control Strategies, Skills, and Harnesses for LLM Agents.
This repository collects must-read papers on LLM-based agents: language models equipped with planning, memory, tool use, and multi-agent coordination to pursue goals over long horizons. Papers follow the taxonomy of the accompanying survey, covering the core components of an agent, the environments and applications they are deployed in, and the cross-cutting concerns of evaluation and safety. Each entry links to the paper and, where an official implementation exists, to its code.
This is a curated, continuously updated selection rather than a copy of the survey's bibliography. Sections are collapsed by default; click Show N papers to expand.
Legend: ⭐ = Starter Kit pick (read these first) ·
[code]= link to an official implementation.
⭐ Starter Kit
New to the area? These ten papers are a good place to start.
| # | Paper | Area | Why start here |
|---|---|---|---|
| 1 | ReAct: Synergizing Reasoning and Acting | Planning | The template for the modern agent loop: interleave reasoning with actions. |
| 2 | Reflexion: Verbal Reinforcement Learning | Planning | Self-reflection stored in memory as a gradient-free improvement loop. |
| 3 | Toolformer: LMs Can Teach Themselves to Use Tools | Tool Use | The seminal self-supervised tool-use paper. |
| 4 | Generative Agents: Interactive Simulacra | Multi-Agent | Memory + reflection at population scale; the canonical agent-memory design. |
| 5 | Voyager: An Open-Ended Embodied Agent | Memory / Env. | Lifelong learning via a growing library of executable skills. |
| 6 | Cognitive Architectures for Language Agents (CoALA) | Foundations | The vocabulary (memory, action space, decision loop) this list is organized around. |
| 7 | A Survey on LLM-based Autonomous Agents | Survey | The canonical general survey of the field. |
| 8 | LLM-based Multi-Agents: A Survey | Multi-Agent | The standard reference for the multi-agent branch. |
| 9 | AgentBench: Evaluating LLMs as Agents | Evaluation | The standard cross-environment agent benchmark. |
| 10 | Not what you've signed up for (Indirect Prompt Injection) | Safety | The founding paper of the agent-security threat model. |
🔥 10 to Watch (2026)
Fresh 2026 work that is already drawing attention.
Contents
- ⭐ Starter Kit
- 🔥 10 to Watch (2026)
- 🧭 Background
- 🧱 Part I: Core Components
- 🌍 Part II: Agents in Context
- ⚖️ Part III: Cross-Cutting Concerns
🧭 Background
📚 Surveys & Position Papers (62)
Corresponds to §1-§3 (Introduction, Background, Taxonomy).
Show 62 papers
A Survey on Large Language Model based Autonomous Agents (Wang et al., arXiv 2023) - The canonical, most-cited general-purpose LLM-agent survey. ⭐ [code]
The Rise and Potential of Large Language Model Based Agents: A Survey (Xi et al., arXiv 2023) - Co-foundational with Wang et al. 2023 as one of the two seminal general surveys. [code]
Cognitive Architectures for Language Agents (Sumers et al., TMLR 2023) - The most widely adopted conceptual/architectural vocabulary for describing LLM agents. ⭐ [code]
ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., ICLR 2023) - The single most-cited technical precursor of modern LLM agents. ⭐ [code]
Reflexion: Language Agents with Verbal Reinforcement Learning (Shinn et al., NeurIPS 2023) - Established the 'self-reflection + memory' loop as an alternative to gradient-based RL for agent self-improvement. ⭐ [code]
Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al., NeurIPS 2023) - Seminal tool-use paper anchoring the 'tool augmentation' pillar of LLM-agent taxonomies. ⭐
Generative Agents: Interactive Simulacra of Human Behavior (Park et al., arXiv 2023) - Foundational demonstration of LLM-driven agent societies/simulation. ⭐ [code]
Voyager: An Open-Ended Embodied Agent with Large Language Models (Wang et al., TMLR 2023) - Seminal example of embodied, code-skill-based, lifelong-learning LLM agents. ⭐ [code]
HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face (Shen et al., NeurIPS 2023) - Foundational example of the 'LLM-as-orchestrator-of-tools/models' agent pattern. [code]
MRKL Systems: A Modular, Neuro-Symbolic Architecture that Combines Large Language Models, External Knowledge Sources and Discrete Reasoning (Karpas et al., arXiv 2022) - Earliest widely-cited neuro-symbolic precursor to LLM tool-use/agent architectures.
Agent AI: Surveying the Horizons of Multimodal Interaction (Durante et al., arXiv 2024) - Broadens the LLM-agent survey landscape to multimodal/embodied agent AI.
Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents (Zhang et al., arXiv 2023) - Bridges the reasoning (CoT) literature and agent literature. [code]
Large Language Model based Multi-Agents: A Survey of Progress and Challenges (Guo et al., IJCAI 2024) - The standard reference survey specifically for the multi-agent branch of LLM agents. ⭐ [code]
Understanding the Planning of LLM Agents: A Survey (Huang et al., arXiv 2024) - Fills the planning-specific gap in the foundational-survey landscape.
Tool Learning with Large Language Models: A Survey (Qu et al., arXiv 2024) - The definitive survey for the tool-use pillar of LLM agents. [code]
A Survey on the Memory Mechanism of Large Language Model based Agents (Zhang et al., arXiv 2024) - The standard survey for the memory subsystem of LLM agents. [code]
Agentic Large Language Models, a Survey (Plaat et al., arXiv 2025) - A recent, widely-referenced general survey with a compact reasoning/acting/interacting taxonomy.
Large Language Model Agent: A Survey on Methodology, Applications and Challenges (Luo et al., arXiv 2025) - One of the most comprehensive and recent (2025) general surveys. [code]
Fully Autonomous AI Agents Should Not Be Developed (Mitchell et al., arXiv 2025) - A prominent, widely-discussed dissenting position paper on agent autonomy.
AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (Sapkota et al., arXiv 2025) - Provides terminological/conceptual disambiguation increasingly cited given inconsistent usage in the field.
LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey (Zou et al., arXiv 2025) - Covers the human-agent cooperation branch identified in the earliest foundational surveys. [code]
Levels of Autonomy for AI Agents (Feng et al., arXiv 2025) - Offers a widely-cited operational framework for comparing autonomy across LLM-agent systems.
Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems (Liu et al., arXiv 2025) - A 48-author landmark survey organizing the field around brain-inspired cognitive modules, self-evolution, collective intelligence, and safety.
The Landscape of Agentic Reinforcement Learning for LLMs: A Survey (Zhang et al., arXiv 2025) - The canonical survey of agentic RL: training LLMs as decision-making agents rather than passive generators.
A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems (Fang et al., arXiv 2025) - Surveys techniques by which agents optimize their own components from interaction data, bridging foundation models and lifelong agentic systems.
Deep Research Agents: A Systematic Examination and Roadmap (Huang et al., arXiv 2025) - First systematic survey of long-horizon autonomous research agents (search, tool use, report synthesis).
A Survey of AI Agent Protocols (Yang et al., arXiv 2025) - Maps the emerging protocol layer (MCP, A2A, and successors) and proposes evaluation dimensions for agent interoperability standards.
Agentic Reasoning for Large Language Models (Wei et al., arXiv 2026) - Survey organizing agentic reasoning into single-agent, self-evolving, and multi-agent layers, bridging in-context reasoning and post-training. [code]
Memory for Autonomous LLM Agents: Mechanisms, Evaluation, and Emerging Frontiers (Du et al., arXiv 2026) - Frames agent memory as a write-manage-read loop with a taxonomy over mechanisms, benchmarks, and applications.
Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations (Jiang et al., arXiv 2026) - Taxonomizes agent memory structures and empirically exposes benchmark saturation and metric-validity issues across systems.
Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systems (Qi et al., arXiv 2026) - Proposes a unified 'LIFE' framework (foundation, integrate, find faults, evolve) for multi-agent collaboration, failure attribution, and self-evolution.
A Technical Taxonomy of LLM Agent Communication Protocols (Sander et al., arXiv 2026) - Analyzes nine open agent-to-agent protocols across five dimensions and predicts convergence toward a federated protocol stack.
Bridging the Agent-World Gap: Text World Models for LLM-based Agents (Li et al., arXiv 2026) - Systematizes text world models (LLM-as-WM vs code-as-WM) that give agents explicit environment prediction for planning and verification. [code]
Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agents (Lahjouji et al., arXiv 2026) - Data-centric survey organizing agent-privacy research by data surface rather than attack type, mapping governance gaps.
Self-Improvements in Modern Agentic Systems: A Survey (Ren et al., arXiv 2026) - Frames a modern agent as a foundation model plus an operational scaffold, and organizes self-improvement by what gets updated (weights or scaffold) and which signal drives the change. [code]
Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries (Li et al., arXiv 2026) - Surveys 124 papers on evolving skill libraries as lifecycle-managed artifact stores, arguing that admission and repair, not acquisition, are the decisive stages.
From Question Answering to Task Completion: A Survey on Agent System and Harness Design (Guo et al., arXiv 2026) - Reads agents through a model-versus-harness lens, decomposing the harness into six runtime responsibilities and asking where the performance bottleneck actually sits. [code]
Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directions (Jing et al., arXiv 2026) - Position paper recasting prompt injection, tool misuse, and memory poisoning as one structural problem: missing isolation boundaries across five agent interfaces.
Always-On Agents: A Survey of Persistent Memory, State, and Governance in LLM Agents (Ding et al., arXiv 2026) - Surveys 435 works on agents with durable cross-session state, finding the literature favours accumulation and retrieval while neglecting governance and recovery.
Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (Zhou et al., arXiv 2026) - Frames the progression of agents as externalization, with capability migrating out of weights into memory, skills, protocols, and harness infrastructure.
SoK: Agentic Skills -- Beyond Tool Use in LLM Agents (Jiang et al., arXiv 2026) - Systematization of agentic skills as reusable callable procedures, drawing the line between a skill and an atomic tool call.
From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms (Luo et al., arXiv 2026) - Surveys agent memory through a three-stage evolution from storage to reflection to experience, driven by consistency, dynamics, and continual learning. [code]
LLM agents security duality: a comprehensive survey of self-security and empowered cybersecurity (Xu et al., arXiv 2026) - A survey that jointly organizes the security threats to LLM agents and their mitigations under a threat-source taxonomy alongside a framework mapping agent capabilities onto the full cyber offense-defense lifecycle.
Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application (Li et al., arXiv 2026) - This survey organizes research on environments for LLM-based agents around an engineering lifecycle of modeling, synthesis, evaluation, and application, proposing an eight-attribute categorization of environments.
Toward Efficient Agents: Memory, Tool learning, and Planning (Yang et al., arXiv 2026) - A survey that reviews efficiency in LLM agent systems across three components (memory, tool learning, and planning), framing effectiveness versus cost as a Pareto frontier.
Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agents (Arunkumar V et al., arXiv 2026) - A survey that proposes a unified six-component taxonomy (Perception, Brain, Planning, Action, Tool Use, Collaboration) for LLM agents and reviews their architectures, operating environments and evaluation practice, closing on open problems such as hallucinated actions, infinite loops and prompt injection.
A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecycle (Lin et al., arXiv 2026) - Surveys security threats to long-term memory in LLM agents by organizing attacks, defenses, and governance across six memory-lifecycle phases and four security objectives, advocating a "Verifiable Memory Governance" framework.
Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunities (Oh et al., arXiv 2026) - Argues uncertainty quantification has to move off single-turn QA and onto interactive agents, giving a general formulation plus the four challenges specific to agents: estimator choice, heterogeneous entities, uncertainty dynamics, and the absence of fine-grained benchmarks.
Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges (Motger et al., arXiv 2026) - Reviews 141 multi-agent debate studies and finds the field has quietly settled on static, fully connected topologies with voting, adopted by convention rather than by controlled comparison.
Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents (Albayaydh et al., arXiv 2026) - Synthesizes 27 evaluation papers across 19 benchmarks into six failure clusters, finding that failures compound nonlinearly with task length and that added scaffolding does not reliably improve reliability.
How Agents Ask for Permission: User Permissions for AI Agents, from Interfaces to Enforcement (Michael et al., arXiv 2026) - Surveys 21 agent permission proposals against five commercial agents, taxonomizing how user-level policies get specified, derived from user input, and enforced at run time, and where gaps remain.
Blockchain Empowered Trustworthy Agent Networks: Foundations, Taxonomy, and Future Directions (Zhu et al., arXiv 2026) - Surveys the road from classical multi-agent systems to open agent networks over 1980 to 2026, arguing that the trust problem moves to the network level once agents owned by different parties transact, and that single-agent safety mechanisms cannot reach it.
Software Engineering for and with GUI Agent (Yu et al., arXiv 2026) - Reviews 336 GUI-agent papers from January 2018 to April 2026 and finds architectures converging on modular perceive-reason-act loops while recovery, human escalation, safety enforcement and auditability stay underdeveloped, with evaluation still centered on task success and hard to compare across protocols.
Agent Safety Should Be a Runtime Contract (Ng et al., arXiv 2026) - Position paper putting agent safety in the harness as a runtime contract with a preventive face (sandboxes, permission gates, monitors) and an evidential face (test runs, log captures, file diffs), grounded in 52 documented incidents, a trajectory-schema audit of 12 public agent systems, and a title-level audit of all 28,560 NeurIPS, ICML and ICLR 2023-2025 papers showing a pooled 8-12x imbalance between training-time and deployment-time publication.
Information Retrieval Misses the Mark for LLM Agents (Sun et al., SSRN 2026) - Position paper answering the "RAG is dead, agents just need grep" reading of deployed practice: it grants that the substrate has shifted but argues the mismatch is structural across five dimensions, since IR's assumed corpus, input, objective, episode and retrievable all sit wrong for an agent that plans, browses, calls tools and decides whether to keep searching, and it tests the gap by holding the agent fixed while swapping BM25, vector, grep, hybrid and closed-book retrieval on HotpotQA-distractor and 2WikiMultihopQA, arguing for retrieval recast as a state-conditioned evidence-acquisition policy.
Terminal Agents: A Survey of AI Agents in Command-Line Environments (Bin et al., arXiv 2026) - Treats the terminal, rather than the task domain, as the organizing lens: agents whose progress-bearing loop runs through command execution and textual feedback, mapped across architecture, competence acquisition and evaluation on a seven-dimensional competence profile, with the survey's own fixed-condition diagnostics showing that benchmark families expose different process signals and that matched system comparisons come out benchmark-dependent, which limits how far any result can be attributed to a single component.
Autonomous Research Agents: A Survey of AI Scientists and the Verification Gap (Ding et al., arXiv 2026) - Codes 26 of 125 screened AI-scientist systems along seven audit dimensions and finds the bottleneck has moved from capability to checkability: 83 per cent of the 24 runnable systems release code, but only 38 per cent release seeds or execution traces and only 38 per cent report any novelty verification, and among nine closed-loop systems seven are mechanical reruns with no externally validated in-loop oracle anywhere in the corpus.
Jev in the Wild: A Data-Driven Analysis of the Jev Model's Functionality, Applications and Ecosystem (Ling et al., arXiv 2026) - Labels 2,170 public GitHub projects that use Jev, collected one week after its release, and finds it used far more to judge inputs than to choose actions: attribute judgment appears in 77% of projects and action selection in 31%, though action selection is about half the labels in interface-agent and simulation-and-control projects, and routing and interface agents are 19.6% of the projects but receive 63% of the stars; an LLM agent decided which repositories count and labeled each one, with a second agent reviewing, and no comparison with human labels is reported.
Why Agentic Workflows Fail: A Taxonomy Derived from Literature and Empirical Data (Li et al., arXiv 2026) - Codes agentic-workflow failures along four dimensions, responsible source (agent, external system or both), mechanism among 65 types, workflow stage and agent capability, and applies the scheme to 397 failure records from papers and 1,196 from GitHub issues and pull requests in 229 repositories: GitHub failures concentrate in actuation during execution, 64.8% of records against 14.1% in the literature, and 61% of those are not the agent's fault, such as a failing external API; 68 of 69 records from later papers fit the existing types, and the counts are not prevalence estimates.
Distilling Agentic Systems: A Roadmap across Models, Artifacts, and Harnesses (Luo et al., arXiv 2026) - Widens distillation from a teacher model to a teacher agent and sorts the literature by where the student keeps what it learned: in the weights, in reusable artifacts such as memories, skills and tools, in the execution harness through workflows, routing and verification, or across these, as when a textual workflow is compiled into a controller or an artifact becomes an enforceable gate; it asks each transfer claim for intrinsic, causal and end-to-end evidence and finds no substrate preferable in general, though each method is filed under a single primary substrate.
Agentic Societies Need a Social Harness (Chugh et al., arXiv 2026) - Argues that agents acting for different people need a social harness that governs the messages between them, on top of each agent's own harness, and supports the case with OpenClaw agents scheduling university meetings, on GPT-5.4 alone or with one agent on Claude Opus 4.8: over ten runs per setting, isolated per-peer sessions make the mixed setup fail every group meeting with more than one teaching assistant, shared sessions lift it from 0% to 90% with seven, and relayed false cancellations succeed in 100% of GPT-5.4 runs; the proposed five-layer stack, from verifiable identity to social institutions, is not built or costed.
Harness Engineering for Language Agents: The Harness Layer as Control, Agency, and Runtime (He et al., Preprints 2026) - Position paper that splits the layer around the model into control (instructions, repository maps, tests, permissions), agency (the tools, interfaces and roles the model acts through) and runtime (memory, checkpoints, retries, budgets), and proposes HarnessCard, a template for reporting an agent's harness next to its model. In its audit of 63 works, the 38 papers and benchmarks most often foreground observability, interfaces and feedback, while the 25 engineering notes, protocols and articles foreground runtime, control and governance. The authors call the audit a visibility check over a selected set with interpretive tags, not a field-wide count, rely substantially on engineering notes from a few frontier labs, and run no experiment that separates model effects from harness effects.
🏗️ Agent Architectures & Frameworks (54)
Corresponds to §2 (Background) and the running examples throughout.
Show 54 papers
Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions (Yang et al., arXiv 2023) - Only peer-reviewed-adjacent empirical study of the widely-influential (but paper-less) AutoGPT autonomous-agent design pattern. [code]
AgentBench: Evaluating LLMs as Agents (Liu et al., ICLR 2024) - The standard reference benchmark for measuring general single-agent capability across heterogeneous environments. ⭐ [code]
WebArena: A Realistic Web Environment for Building Autonomous Agents (Zhou et al., ICLR 2024) - The de facto standard testbed for web-browsing single-agent architectures. [code]
Gorilla: Large Language Model Connected with Massive APIs (Patil et al., NeurIPS 2024) - Key single-agent tool-use paper demonstrating that fine-tuning plus retrieval can make an agent reliably invoke large real-world API catalogs. [code]
ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Models (Xu et al., arXiv 2023) - Influential efficiency-oriented alternative to the ReAct loop, illustrating the plan-then-execute vs. interleaved architectural design axis. [code]
Tree of Thoughts: Deliberate Problem Solving with Large Language Models (Yao et al., NeurIPS 2023) - A core deliberate-search reasoning architecture underpinning later single-agent planning/search frameworks like LATS. [code]
WebGPT: Browser-assisted question-answering with human feedback (Nakano et al., arXiv 2021) - Early pre-ChatGPT precursor to modern LLM web agents, establishing the browsing-tool-use-plus-human-feedback pattern.
SELF-REFINE: Iterative Refinement with Self-Feedback (Madaan et al., NeurIPS 2023) - A minimal, widely-adopted single-agent self-improvement loop that is reused as a subroutine inside many larger agent architectures. [code]
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (Yang et al., NeurIPS 2024) - Demonstrates how interface design materially changes single-agent capability, now standard in coding-agent design. [code]
Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models (Zhou et al., ICML 2024) - Represents the state-of-the-art convergence of search-based planning with the ReAct/Reflexion lineage. [code]
Executable Code Actions Elicit Better LLM Agents (Wang et al., ICML 2024) - Established 'code-as-action' as a leading alternative action-space design for single agents. [code]
Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents (Wang et al., NeurIPS 2023) - Key single-agent planning architecture for open-world/embodied tasks. [code]
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement (Wu et al., arXiv 2024) - A leading recent example of a general-purpose, self-improving OS-level single agent, extending AutoGPT-style autonomy to real computer environments. [code]
AppAgent: Multimodal Agents as Smartphone Users (Zhang et al., CHI 2025) - Representative recent single-agent architecture extending the ReAct/tool-use paradigm to GUI/mobile control. [code]
The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey (Masterman et al., arXiv 2024) - A survey specifically scoped to agent architecture design patterns, directly relevant to categorizing single-agent frameworks.
MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework (Hong et al., ICLR 2024) - Widely cited framework showing how single-agent role/procedure templates improve reliability, marking the transition point between single- and multi-agent framework design. [code]
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities (Comanici et al., arXiv 2025) - Frontier model report that treats agentic tool use and computer operation as headline capabilities.
Kimi K2: Open Agentic Intelligence (Kimi Team, arXiv 2025) - Flagship open model built explicitly around agentic post-training at scale. [code]
GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization (Liang et al., arXiv 2026) - Token-efficient self-evolving agent that accumulates contextual experience; one of the most-starred 2026 agent frameworks. [code]
Orchestral AI: A Framework for Agent Orchestration (Roman et al., arXiv 2026) - Framework for composing and orchestrating specialized agents behind a single interface. [code]
AgentArk: Distilling Multi-Agent Intelligence into a Single LLM Agent (Luo et al., arXiv 2026) - Distills multi-agent intelligence into a single LLM agent, retaining collaboration gains at lower cost. [code]
The Interplay of Harness Design and Post-Training in LLM Agents (Kim et al., arXiv 2026) - Shows harness design and post-training interact, so harness-aware post-training improves both in-distribution and out-of-distribution performance.
Next-Generation Agentic Reinforcement Learning Systems Enable Self-Evolving Agents (Ran Yan et al., arXiv 2026) - Argues the agentic RL systems stack, not the algorithm, is what enables agents that revise their own components rather than only their weights.
From Atomic Actions to Standard Operating Procedures: Iterative Tool Optimization for Self-Evolving LLM Agents (Ding et al., arXiv 2026) - Synthesizes recurring action sequences from execution traces into callable higher-order procedures, then merges, evaluates, and prunes the resulting toolset.
Self-Evolving World Models for LLM Agent Planning (Zhang et al., arXiv 2026) - Refines the agent's internal world model at test time while holding the executor fixed, tightening the plan-and-simulate loop.
Scaling Self-Evolving Agents via Parametric Memory (Ren et al., arXiv 2026) - Moves self-evolution out of the context window and into parameters, so accumulated experience scales without growing the prompt.
Inside the Scaffold: A Source-Code Taxonomy of Coding Agent Architectures (Rombaut et al., arXiv 2026) - Derives a source-code taxonomy of 13 open coding-agent scaffolds, identifying five composable loop primitives that systems recombine.
Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses (Lin et al., arXiv 2026) - Automatically evolves a coding-agent harness from observability signals, raising Terminal-Bench 2 from 69.7 to 77.0.
AgentFactory: A Self-Evolving Framework Through Executable Subagent Accumulation and Reuse (Zhang et al., arXiv 2026) - Accumulates capability by saving successful solutions as reusable executable subagent code rather than textual prompts, refined by execution feedback. [code]
CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended Discovery (Qu et al., COLM 2026) - Replaces hard-coded exploration rules with long-running agents that explore, reflect and collaborate through shared persistent memory, under isolated workspaces and evaluator separation; sets state of the art on 10 optimization tasks at 3 to 10 times the improvement rate per evaluation, and four co-evolving agents take Anthropic's kernel task from 1363 to 1103 cycles. [code]
LogicHunter: Testing LLM Agent Frameworks with an Agentic Oracle (Long et al., arXiv 2026) - Proposes LogicHunter, a specification-driven fuzzing framework paired with a ReAct-based agentic oracle that retrieves documentation, navigates source code and inspects runtime state; on three widely deployed agent frameworks it found 40 previously unknown bugs, 30 confirmed and 26 fixed, with the oracle at 91.17% precision against 29.27% for the best passive approach.
AgentFlow: Building Agent Dependency Graphs for Static Analysis of Agent Programs (Wang et al., arXiv 2026) - Introduces AgentFlow, a static analysis framework that recovers agent dependencies from LLM agent source code by constructing a framework-agnostic Agent Dependency Graph whose typed nodes cover agents, prompts, models, capabilities, memory states and control policies; across 5,399 real-world agent programs it surfaces 238 taint-style prompt-to-tool risks.
SEAGym: An Evaluation Environment for Self-Evolving LLM Agents (Zheng et al., arXiv 2026) - Introduces SEAGym, an evaluation environment for self-evolving LLM agents that measures agent-harness modifications across training, validation, held-out test, replay, and cost dimensions rather than a single task score.
Harness-MU: A Safe, Governed, and Effective Harness for Multi-User LLM Agents (Fan et al., arXiv 2026) - Proposes Harness-MU, a model-agnostic, zero-tuning harness that enforces multi-principal access control for LLM agents through deterministic execution hooks rather than model-internal safeguards. [code]
Co-Evolving Skill Generation and Policy Optimization (Zhang et al., arXiv 2026) - Proposes an online framework for skill-augmented language agents that estimates each candidate skill's context-dependent marginal utility via matched comparison groups (retrieved skills with and without the candidate) to filter out ineffective or harmful skills before they are stored, within the standard rollout budget, and trains the policy itself to generate skills.
DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrations (Che et al., arXiv 2026) - DemoEvolve uses human expert demonstration trajectories to guide the search in agentic harness evolution, addressing sparse-reward instability in complex stochastic environments like Balatro where self-practice alone struggles.
Self-Evolving Software Agents (Robol et al., arXiv 2026) - Proposes an architecture combining BDI (Belief-Desire-Intention) reasoning with large language models in which an automated evolution module elicits new requirements from experience and synthesizes corresponding design and code.
Codified Context: Infrastructure for AI Agents in a Complex Codebase (Vasilopoulos et al., arXiv 2026) - Proposes a three-component infrastructure (a conventions constitution, 19 specialized agents, and a knowledge base of 34 specification documents) to give LLM coding assistants persistent context. [code]
LLM-as-a-Verifier: A General-Purpose Verification Framework (Kwok et al., arXiv 2026) - Treats verification as its own scaling axis: continuous scores from scoring-token logits, scaled by granularity, repeated evaluation and criteria decomposition, reach 86.5% on Terminal-Bench V2 without extra training. [code]
Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learning (Hu et al., arXiv 2026) - Shows a compact PyTorch-native agentic RL trainer, small enough for a researcher or coding assistant to read end to end, stays statistically comparable to a Megatron-based stack. [code]
Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing (Dhakal et al., arXiv 2026) - Default coding CLI agents already solve a large share of the 104-task XBOW benchmark, and repeated plain-agent runs can match published harness architectures under matched models.
Argus: A General-Purpose Agentic Runtime for Long-Horizon Reasoning (Li et al., arXiv 2026) - A persistent runtime in which Manager, Planner, Engineer and Reviewer run bounded missions over durable project state, admitting memories, skills and verifiers only after role-owned review; weights stay fixed, and 78% on SWE-Bench Pro against 59% for a direct baseline comes from runtime state alone.
Architectural Implications of Agentic AI Workflows (Yang et al., arXiv 2026) - Characterizes agentic workloads as a datacenter problem through a production study at Azure, finding that orchestration and tools put the CPU on the critical path and that fragmented execution strands both CPU and GPU capacity on conventional uniform servers.
Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution (Razzhigaev et al., arXiv 2026) - A coding-agent harness whose tools, prompts, context assembly, and core implementation improve through reviewed commits that become the runtime for later work; reports 86.74% on Terminal-Bench 2.1 and 90.69% on OSWorld-Verified from frozen snapshots, while a separate 161-day deployment keeps evolving live. [code]
The Scaffolding Matters More Than the Interface: A Controlled Comparison of MCP and CLI Tool Use Across Seven Agent Scaffoldings, Five Language Models, and One Software Task (Alier Forment et al., arXiv 2026) - One fixed git task run across seven agent scaffoldings and five language models shows the scaffolding, not the MCP-versus-CLI interface, drives cost: the two scaffoldings with no MCP support completed every run over the CLI and were 5.0x to 28x cheaper than the five MCP-capable scaffoldings on CLI runs alone, thirteen strictly paired MCP-to-CLI ratios span 0.43x to 29x, and 12.9 per cent of the money spent on MCP runs bought no completed work against 2.2 per cent on CLI runs.
Persistent Recursive Worlds Enable Autonomous Software Evolution (Huang et al., arXiv 2026) - Makes the software project persistent rather than the agent: finite-lived agents propose local changes and only accepted consequences advance the version history, and one run of over 120 hours built a roughly 250k-line Rust C compiler that passed the complete c-testsuite for 44 US dollars in model-token charges.
What Does Multi-Harness RL Learn? Credit Assignment and Portability in Coding Agents (Le et al., arXiv 2026) - Replays frozen records from Aider, OpenHands, Qwen Code and SWE-agent to isolate whether pooling harnesses inside one advantage group buys portable skill, and the evaluation harness dominates everything else: across 24,000 sealed evaluations it moves mean solve rate from 2.14 to 9.27 per cent where the training recipe moves it by 1.16, while cross-harness grouping beats within-harness grouping by 0.25 points on a held-out harness, a confidence interval spanning zero and narrower than each rule's own seed range.
TROVE: Adaptive Agent Skill Orchestration via Trace-Grounded Route Validation and Editing (Wang et al., arXiv 2026) - Treats a planned route as provisional rather than committed: offline it distils evaluated workflow-search traces into atomic and composite skills plus an outcome-conditioned transition graph, and online it keeps the valid continuation, inserts a trace-supported local response, or replaces only the invalid suffix, so runtime evidence buys local repair instead of a broad replan; ablations put most of the offline benefit in the composite skills and most of the efficiency gain in suffix replacement.
SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks (Lin et al., NeurIPS 2023) - Splits the agent into a small fine-tuned model that acts fast and a GPT-4 planner called in only on triggers such as a stalled or invalid action, and beats SayCan, ReAct and Reflexion across 30 ScienceWorld task types.
R2V Agent: Teaching SLMs When to Ask for Help (Hemadri et al., arXiv 2026) - Routes per step instead of per query, since difficulty shifts mid-trajectory: a Brier-calibrated router escalates from a distilled small model to a teacher LLM only when residual failure risk is high, lifting TextWorld from 64.6% to 98.2% success at 41.7% escalation. [code]
REFLEX with Jev for Efficient Selective Control in LLM Agents (Wu and Lim, arXiv 2026) - Lets Jev, a model that answers with a typed decision instead of text, make an agent's bounded choices and calls a strong LLM only when confidence is low or text is needed: 95% success with 72.7% fewer strong-model calls on a frozen 100-task benchmark, but little gain over a cheap generative cascade on BFCL and τ-style tasks.
What Does a Harness Buy? Tokens, Mostly (Liu and Han, arXiv 2026) - Runs five models through Claude Code, mini-SWE-agent and OpenCode on SWE-bench Verified and reruns each configuration to measure the noise: on 447 tasks Claude Code and mini-SWE-agent come out equivalent within 5 points on both Qwen models run there, on the 45 hardest tasks swapping the harness flips 13% of tasks, the same share as a rerun, and the one effect above the noise is OpenCode trailing by up to 9 points, while cost per task differs up to threefold, set by a fixed preamble of 16k tokens in Claude Code against under a thousand in mini-SWE-agent; one benchmark only. [code]
When Sub-Agents Work in Parallel: The Promises and Pitfalls of Dynamic Concurrency in Long-Horizon Coding Tasks (Li et al., arXiv 2026) - Switches the built-in dynamic concurrency of Codex (GPT-5.4), Claude Code (Claude Opus 5) and Kimi Code (Kimi K3), in which the main agent decides at run time whether to spawn parallel sub-agents, on and off across 354 coding tasks: pass rates move between an 11.9-point drop and a 2.3-point gain while token use grows 1.41 to 3.31 times, Claude Code falls from 83.0% to 59.0% on SWE-bench Verified yet rises from 7.1% to 21.4% on the long-horizon LoopsBench, and 33.2% of 804 annotated failures are shared-state and merge problems; each configuration ran once per task.
Ask Your Advisor: Learning When to Consult Stronger Models in Agentic Systems (Sowietzki et al., arXiv 2026) - Learns when a cheap executor should consult a stronger advisor, choosing to stay silent or consult at each workflow contact point from a compact state abstraction, with Monte Carlo control on cost-penalized outcomes: on Magentic-One with gemini-3-flash-preview executing and gemini-3.1-pro-preview advising, it reaches 48.0% on SWE-bench Verified-Mini at $22.09 against 18.0% for the executor alone and 62.0% at $45.90 for the advisor alone, and beats RouteLLM (30.0% at $25.03) and random consultation at a similar volume (28.0%), but only ties a heuristic on Terminal-Bench at 34.7%, with one model pair, one framework and 50-task test sets. [code]
🧱 Part I: Core Components
🧠 Planning & Reasoning (55)
Corresponds to §4 (Planning and Reasoning).
Show 55 papers
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., NeurIPS 2022) - The foundational technique underlying virtually all LLM-agent reasoning/planning modules; the starting point for the whole CoT/ToT/ReAct lineage.
Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al., ICLR 2023) - Standard inference-time ensembling/verification strategy widely reused inside agent reasoning and planning pipelines.
Large Language Models are Zero-Shot Reasoners (Kojima et al., NeurIPS 2022) - Established that reasoning behavior is latent and promptable zero-shot, a key enabler for general-purpose agent prompting templates. [code]
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (Zhou et al., ICLR 2023) - Early formalization of task decomposition, a core planning primitive reused by nearly all subsequent LLM-agent task planners.
STaR: Bootstrapping Reasoning With Reasoning (Zelikman et al., NeurIPS 2022) - Precursor to the RL-based reasoning-model training paradigm (e.g., DeepSeek-R1, o1) used to instill self-generated reasoning in agents. [code]
Graph of Thoughts: Solving Elaborate Problems with Large Language Models (Besta et al., AAAI 2024) - Extends structured-reasoning/search frameworks used by agents beyond trees, improving quality and cost on complex multi-step tasks. [code]
Reasoning with Language Model is Planning with World Model (Hao et al., EMNLP 2023) - Connects classical planning-as-search (MCTS, world models) with LLM reasoning, directly relevant to agent planning under uncertainty. [code]
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (Gou et al., ICLR 2024) - Bridges self-critique with tool-augmented verification, a key mechanism in modern agent frameworks that ground reflection in external feedback. [code]
Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models (Wang et al., ACL 2023) - Widely-used lightweight plan-then-execute template that many LLM agent planning modules adopt directly. [code]
ADaPT: As-Needed Decomposition and Planning with Language Models (Prasad et al., ACL 2024) - Demonstrates adaptive, execution-conditioned planning that balances plan granularity against agent capability, an important refinement over static plan-and-execute agents. [code]
Self-Discover: Large Language Models Self-Compose Reasoning Structures (Zhou et al., NeurIPS 2024) - Shows LLMs can self-select their own reasoning strategy per task, a meta-reasoning capability central to adaptive agent planning.
Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models (Yang et al., NeurIPS 2024) - Represents reusable, memory-augmented reasoning-structure retrieval, connecting reasoning strategies with agent long-term memory design. [code]
Large Language Models Cannot Self-Correct Reasoning Yet (Huang et al., ICLR 2024) - A widely-cited cautionary/critical result that tempers over-reliance on self-critique loops in agent design, motivating external-feedback-grounded methods.
Training Language Models to Self-Correct via Reinforcement Learning (Kumar et al., ICLR 2025) - Shows self-correction can be made genuinely effective via RL rather than prompting alone, directly informing reasoning-model post-training used in modern agents.
Let's Verify Step by Step (Lightman et al., ICLR 2024) - Establishes process reward models/step-level verification, now a standard component for guiding search and self-critique in reasoning agents. [code]
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (Guo et al., Nature 2025) - Landmark open reasoning-model release demonstrating RL-induced emergent planning/reflection behaviors now underpinning next-generation reasoning agents. [code]
Towards Reasoning in Large Language Models: A Survey (Huang et al., ACL 2023) - One of the earliest and most-cited surveys specifically on LLM reasoning, a natural anchor citation for any LLM-agent survey's reasoning section. [code]
Large Language Models for Planning: A Comprehensive and Systematic Survey (Cao et al., arXiv 2025) - A dedicated, up-to-date survey covering exactly the planning-strategies portion of this sub-topic, ideal for anchoring taxonomy claims. [code]
When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs (Kamoi et al., ACL 2024) - The key critical survey specific to self-critique/self-correction, essential for a balanced, rigorous treatment of reflection methods in an agent survey. [code]
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (Jin et al., arXiv 2025) - Canonical agentic-RL result: the model learns to interleave search-engine calls with its own reasoning from outcome rewards alone. [code]
ReTool: Reinforcement Learning for Strategic Tool Use in LLMs (Feng et al., arXiv 2025) - RL that teaches a reasoning model when and how to invoke a code interpreter mid-derivation.
Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents (Lin et al., arXiv 2026) - Separates 'harness-updating' from 'harness-benefit' in self-evolving agents, finding a non-monotonic benefit curve across model tiers. [code]
Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe (Wu et al., arXiv 2026) - Empirical recipe over reward shaping, model scale, data, and algorithm choices for agentic RL, reaching SOTA on TravelPlanner. [code]
StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction (Xue et al., arXiv 2026) - Jointly trains strategy generation and action execution via sampled trajectory abstractions, improving agentic RL on ALFWorld/WebShop/SciWorld.
The Self-Correction Illusion: LLMs Correct Others but Not Themselves (Chen et al., arXiv 2026) - Shows LLMs correct external claims but not identical self-authored errors, an artifact of chat-template role labels rather than a capability gap.
Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops (Chen et al., arXiv 2026) - Maps the spectrum from bounded self-refinement to autonomous research loops, and where each step of recursive self-improvement currently breaks.
SIRI: Self-Internalizing Reinforcement Learning with Intrinsic Skills for LLM Agent Training (He et al., arXiv 2026) - Reinforcement learning that internalizes discovered skills into the policy rather than leaving them in an external library. [code]
Agentic Chain-of-Thought Steering for Efficient and Controllable LLM Reasoning (Xia et al., arXiv 2026) - Steers agentic chain-of-thought at inference for controllable reasoning length, trading compute against accuracy without retraining. [code]
ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RL (Xie et al., arXiv 2026) - Selective turn memory for agentic RL: prunes the trajectory to act, keeps the trace to learn from.
AgentTether: Graph-Guided Diagnosis and Runtime Intervention for Reliable LLM Agent Operation (Zhao et al., arXiv 2026) - Diagnoses agent failures on a graph of the run and intervenes at runtime, rather than only post-hoc.
Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search (Zhang et al., arXiv 2026) - Replaces tree search in an MLE agent with a gradient-style optimization framework, mapping reasoning to gradients and success memory to momentum. [code]
MAP: A Map-then-Act Paradigm for Long-Horizon Interactive Agent Reasoning (Liu et al., arXiv 2026) - Builds a cognitive map of the environment before acting, separating mapping from action for long-horizon interactive reasoning.
Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs (Hong et al., arXiv 2026) - Reflective test-time planning for embodied agents that learns from its own trials and errors within a single episode. [code]
Agora: Enhancing LLM Agent Reasoning Via Auction-Based Task Allocation (Zhou et al., arXiv 2026) - Proposes Agora, an auction-based orchestration framework in which reasoning steps are treated as tradeable items and expert models and tools bid on them according to rectified competence rather than raw confidence.
Training the Orchestrator: A Supervised Approach to End-to-End PDDL Planning with LLM Agents (Mangannavar et al., arXiv 2026) - Introduces HALO, which supervises a small QLoRA-tuned orchestrator policy on verifier-certified plan-refinement trajectories across 11 PDDL domains.
Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillation (Guo et al., arXiv 2026) - Trains agents on tool outputs and execution results with no reward signal, using each observed environment response twice: the model learns to predict it from the history and the action, and an EMA copy of the model, given the response as hindsight, is distilled into the policy that has to act without it; with Qwen3-8B it reaches 43.2% success on τ-bench Retail against 39.1% for GRPO trained on rewards and 36.8% for SDPO, and 35.71 against 32.14 and 25.00 on AppWorld task goal completion, and without the prediction loss the 4B model's Retail success drops from 38.7% to 29.4%; both backbones are small Qwen3 models.
Retrospective Progress-Aware Self-Refinement for LLM Agent Training (Ma et al., arXiv 2026) - Introduces RePro, a forward-then-reflect rollout framework in which an LLM agent executes actions online and then retrospectively reassesses its step-wise progress conditioned on the completed trajectory and known outcome.
LiTS: A Modular Framework for LLM Tree Search (Li et al., arXiv 2026) - LiTS is a Python framework that decomposes LLM tree search into reusable Policy, Transition, and RewardModel components supporting algorithms like MCTS and BFS, evaluated across MATH500, Crosswords, and MapEval. [code]
Localizing and Correcting Errors for LLM-based Planners (Kumar et al., arXiv 2026) - Proposes Localized In-Context Learning (L-ICL), which localizes constraint violations in LLM-generated plans and injects minimal corrective examples for failing steps.
CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning (Xu et al., arXiv 2026) - Proposes CLEANER, which uses Similarity-Aware Adaptive Rollback to build purified agentic RL trajectories by replacing failed steps with successful self-corrections, improving accuracy on AIME24/25 while using fewer training steps.
VERGE: Formal Refinement and Guidance Engine for Verifiable LLM Reasoning (Singh et al., arXiv 2026) - A neurosymbolic framework that decomposes LLM outputs into atomic claims, formalizes them into first-order logic, and verifies consistency with SMT solvers to iteratively refine answers, using multi-model consensus.
TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning (Qi et al., arXiv 2026) - Travel-planning benchmark graded by a deterministic rule-based evaluator instead of an LLM judge; the strongest of 15 agents returns a fully feasible itinerary on only 46.2% of solvable tasks. [code]
PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning (Fox et al., arXiv 2026) - Keeping the full structured interaction log and searching it with a coding agent adds 18 points on ARC-AGI-3 over a base coding agent, matching specialist harnesses at 4.2-5.8x fewer tokens. [code]
The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation (Men et al., arXiv 2026) - Controlled study of where long-horizon planning ability comes from: CoT state-transition modeling in pre-training generalizes best, suboptimal trajectories hurt disproportionately, and multi-teacher distillation only merges compatible planning patterns.
SearchMaster: Grounded and Regulated Self-Play for Search Agents (Tan et al., arXiv 2026) - Self-play for search agents that names the three ways self-generated data misleads: evidence chains to stop pseudo multi-hop questions, difficulty scored by search depth rather than success rate, and a penalty on opening documents the agent never uses. [code]
R³-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets (Wang et al., arXiv 2026) - Under a compute budget shared across a six-problem suite, an offline oracle built from the same model's matched single-problem response curves matches or beats the model's actual suite score in all 72 cells and is strictly higher in 71, exposing a gap between demonstrated single-problem competence and what models realize under a shared budget; trajectory diagnostics show limited strategy updating. [code]
Second Thought: Reasoning in Parallel as LLM Agents Act and Observe (Sun et al., arXiv 2026) - Forks four auxiliary reasoning branches into the idle window where a ReAct agent is waiting on the environment and merges them back at the observation, lowering turn counts in all nine model-benchmark pairs and main-thread decoding by up to 43% in six of them, with Pass@1 statistically unchanged in seven of nine.
CHIME: Credit-Aware Hierarchical Memory Evolution for Long-Horizon Agentic Planning (Ye et al., arXiv 2026) - Keeps a planning bank separate from an execution bank and attributes each task outcome to the plan, the execution, both or neither before writing anything down, on the argument that final-outcome feedback conflates plan quality with execution error and environment noise; the resulting memory is smaller, its learned values track downstream utility with planning memories worth more than execution ones, and it transfers across backbone models.
Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents (Huang et al., arXiv 2026) - Benchmarks the decision to stop rather than the decision to act, covering instructions that contradict themselves and instructions that contradict what is on screen, and finds execution-biased overcompliance: agents that score well on feasible tasks keep executing under conflicting ones, which an inference-time feasibility check plus action modulation reduces across five agents without costing normal task performance.
Steer, Don't Solve: Training Small Critic Models for Large Code Agents (Gandhi et al., arXiv 2026) - Trains 4B and 8B critic models with SFT and DPO to identify errors in a coding agent's trajectory and give high-level guidance every few steps at inference time, without generating actions; the critics improve SWE-bench Verified resolve rates of six larger agents, by 16.0 points for GLM-4.7-Flash-30B-A3B and 14.4 for GPT-OSS-120B, and where the agent finishes in fewer steps the steering also lowers cost, taking GPT-OSS-20B from $0.07 to $0.03 per example. [code]
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (KnowNo) (Ren et al., CoRL 2023) - Applies conformal prediction to an LLM planner's candidate next steps so the robot asks a human only when more than one step survives the threshold, giving a statistical guarantee on task completion while keeping human help to a minimum.
Real-Time Detection and Repair of LLM Agent Failures (Dubey, arXiv 2026) - Replaces per-step LLM judging, which costs more than the agent, with telemetry monitors at about 200 microseconds a step plus a deterministic recompute check, and rolls flagged runs back to lift task success from 52% to 73%; the monitors must be recalibrated for each deployment. [code]
LLM Value Functions for Credit Assignment in Agentic RL (Zhu et al., arXiv 2026) - Uses a frozen LLM as the critic in agent RL: shown summaries of other rollouts on the same task, and anchored to the group success rate at the start and the observed reward at the end, it scores every intermediate state in one query, and the value differences become per-step advantages that sum to the group-relative return; with Claude Opus 4.6 as critic, Qwen3-8B reaches 26.3% on AppWorld and 20.2% on SWE-bench Verified against 19.8% and 12.0% for GRPO, but the gain depends on the critic, and with Qwen3-235B in that role Qwen3-4B reaches only 9.8% on AppWorld against 24.3% with Opus.
Judged Useless, Queried Anyway: Tool-Using Agents Rarely Turn Their Own Evidence Judgments into Stopping Decisions (Zhang et al., arXiv 2026) - Separates what a tool-using agent concludes about its evidence from what it does next, on HotpotQA with a search source made to fail: the seven agents call the failing source's results useless 97 to 100% of the time, yet after five such judgments in a row unaided Qwen3-8B answers on 1 of 292 questions, and a stopping rule written into the prompt is followed at most partly; a rule enforced by the harness that forces an answer after five useless results raises success over six failure regimes by 0.026 to 0.097 for four open models and Claude Haiku 4.5, with the failures simulated. [code]
Thinking Before Thinking: Scaling Agentic Inference Through Meta-Reasoning (Dahal et al., arXiv 2026) - Separates an agent's control from its work: workers do the task while a controller, in each cycle, updates a compact record of progress, proposes next steps, weighs them against the remaining call budget and dispatches workers, with full outputs kept in memory for reuse; against a direct-control agent with the same workers and budget it scores higher in all 12 model and benchmark pairings, by 3.6 to 4.2 points on average per benchmark, and reaches 71.5% on ProgramBench with GPT-5.5 against 58.0% for Codex and 63.7% for direct control, but with Opus 4.8 at 400 calls it trails direct control, 56.6% to 62.7%.
💾 Memory (60)
Corresponds to §5 (Memory).
Show 60 papers
RET-LLM: Towards a General Read-Write Memory for Large Language Models (Modarressi et al., arXiv 2023) - Early and influential structured/triplet-based read-write memory design, a precursor to graph- and KG-based agent memory systems.
MemoryBank: Enhancing Large Language Models with Long-Term Memory (Zhong et al., AAAI 2024) - One of the first systems to bring a psychologically grounded (human-memory-inspired) forgetting/consolidation mechanism into LLM agent memory. [code]
Augmenting Language Models with Long-Term Memory (Wang et al., NeurIPS 2023) - Key architectural approach to giving the underlying language model itself (not just an agent scaffold) a trainable long-term memory retrieval mechanism. [code]
Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control (Zheng et al., ICLR 2024) - Demonstrates retrieval-augmented episodic-trajectory memory for grounding agent decision-making in complex GUI/computer-control tasks. [code]
ExpeL: LLM Agents Are Experiential Learners (Zhao et al., AAAI 2024) - Influential paradigm for extracting reusable, transferable 'experience' knowledge from an agent's own memory rather than raw episodic replay. [code]
Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading (MemWalker) (Chen et al., arXiv 2023) - Influential tree-structured (hierarchical) memory-navigation approach bridging long-context modeling and agentic memory retrieval.
MemGPT: Towards LLMs as Operating Systems (Packer et al., COLM 2024) - One of the most widely cited and productized (Letta) architectures for tiered, self-managed long-term memory in LLM agents. [code]
Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memory (Liu et al., arXiv 2023) - Addresses inconsistent-reasoning pitfalls of naive memory recall by storing reasoning traces rather than raw text, influencing later 'reflective retrieval' memory designs.
Evaluating Very Long-Term Conversational Memory of LLM Agents (Maharana et al., ACL 2024) - The standard benchmark used to evaluate and compare long-term conversational memory systems for LLM agents (used by Mem0, MIRIX, A-MEM, etc.). [code]
Larimar: Large Language Models with Episodic Memory Control (Das et al., ICML 2024) - Representative of the architecture-level (rather than prompt-level) approach to giving LLMs an editable, fast-updating episodic memory. [code]
HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (Gutiérrez et al., NeurIPS 2024) - A highly influential neuro-inspired long-term memory/RAG framework bridging knowledge-graph retrieval and agent memory, widely adopted as a strong baseline. [code]
On the Structural Memory of LLM Agents (Zeng et al., arXiv 2024) - Provides a controlled empirical comparison of memory-structuring choices, useful evidence base for a survey's discussion of design trade-offs in agent memory.
A-MEM: Agentic Memory for LLM Agents (Xu et al., arXiv 2025) - Representative of the newest generation of dynamically self-organizing (graph/note-linking) long-term memory systems for LLM agents. [code]
From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs (Wu et al., arXiv 2025) - A recent, comprehensive sub-topic survey giving a psychology-grounded taxonomy that a broader LLM-agent survey can cite for organizing the memory literature.
Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory (Chhikara et al., arXiv 2025) - A leading production-oriented, widely-deployed long-term memory system for LLM agents, frequently used as a state-of-the-art comparison point. [code]
MIRIX: Multi-Agent Memory System for LLM-Based Agents (Wang et al., arXiv 2025) - Illustrates the current frontier trend of multi-agent, multi-type (multimodal) memory architectures for LLM-based agents. [code]
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (Wu et al., ICLR 2025) - Decomposes long-term interactive memory into five separately testable abilities. [code]
ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory (Ouyang et al., arXiv 2025) - Distills strategy-level memory from both successful and failed trajectories so an agent self-evolves over a task stream.
Memory OS of AI Agent (Kang et al., EMNLP 2025) - Applies operating-system memory management (STM/MTM/LPM tiers with heat-based promotion and segmented paging) to agent memory, with strong LoCoMo gains. [code]
Zep: A Temporal Knowledge Graph Architecture for Agent Memory (Rasmussen et al., arXiv 2025) - A bi-temporal knowledge-graph memory engine (Graphiti) that dynamically fuses chat and business data, surpassing MemGPT on DMR and LongMemEval; a standard production-memory reference. [code]
What Deserves Memory: Adaptive Memory Distillation for LLM Agents (Ma et al., ACL 2026) - Self-organizing episodic memory (Nemori) that segments dialogue by event boundaries and distills semantics via a predict-then-calibrate loop, cutting build cost while improving temporal reasoning. [code]
MemOS: A Memory OS for AI System (Li et al., arXiv 2025) - Promotes memory to a first-class resource (MemCube), unifying parametric, activation, and plaintext memory under one scheduling-and-governance operating system. [code]
G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems (Zhang et al., NeurIPS 2025) - Organization-theory-inspired three-tier graph (insight/query/interaction) that stores collaboration trajectories, a memory design tailored to multi-agent systems. [code]
Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models (Zhang et al., ICLR 2026) - Evolves the context itself as a playbook via generate-reflect-curate delta operations, countering brevity bias and context collapse for self-improving agents. [code]
SimpleMem: Efficient Lifelong Memory for LLM Agents (Liu et al., arXiv 2026) - Semantic-compression lifelong memory (structured compression, online synthesis, intent-aware retrieval) cutting inference tokens up to 30x. [code]
PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agents (Yang et al., arXiv 2026) - Cognitive-science-inspired knowledge-graph memory pluggable across tasks without redesign. [code]
Memanto: Typed Semantic Memory with Information-Theoretic Retrieval for Long-Horizon Agents (Abtahi et al., arXiv 2026) - Typed 13-category memory schema with information-theoretic single-query retrieval, SOTA on LongMemEval and LoCoMo.
MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems (Lee et al., arXiv 2026) - 15.6K-QA benchmark (up to 1.8M-token contexts) showing memory agents struggle under interfering, frequently-updated facts. [code]
What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents (Colaco et al., arXiv 2026) - Gives context compaction a rate-distortion formulation, making what to keep and what to forget an explicit distortion budget rather than a heuristic.
AutoMem: Automated Learning of Memory as a Cognitive Skill (Wu et al., arXiv 2026) - Treats memory management itself as a skill the agent learns, instead of a fixed retrieval policy written by the harness. [code]
TokenPilot: Cache-Efficient Context Management for LLM Agents (Xu et al., arXiv 2026) - Manages context with the prompt cache in mind, preserving cache continuity under eviction to cut cost. [code]
Self-GC: Self-Governing Context for Long-Horizon LLM Agents (Hao et al., arXiv 2026) - Lets the agent govern its own context over long horizons, deciding when to compact rather than compacting on a fixed schedule.
MemSyco-Bench: Benchmarking Sycophancy in Agent Memory (Xiang et al., arXiv 2026) - Benchmarks sycophancy in agent memory: whether stored beliefs bend to user pressure across sessions. [code]
Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory (Zou et al., arXiv 2026) - Gives agent memory a rate-distortion formulation, keeping the decision rather than the description under an explicit distortion budget.
LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues (Wu et al., arXiv 2026) - Long-term agent memory benchmark aimed at the experienced-colleague setting, past short-context question answering.
Experience Compression Spectrum: Unifying Memory, Skills, and Rules in LLM Agents (Zhang et al., arXiv 2026) - Unifies memory, skills, and rules as points on one experience-compression spectrum rather than separate mechanisms.
PLACEMEM: Toward a Compute-Aware Memory Plane for Lifelong Agents (Ganguly et al., arXiv 2026) - Proposes PLACEMEM, a memory plane for lifelong agents built on versioned capsules that bind semantics, provenance, and validity under a correction-aware identity.
COMFYCLAW: Self-Evolving Skill Harnesses for Image Generation Workflows (Li et al., arXiv 2026) - Proposes an agentic framework that formulates image-generation workflow construction as typed graph editing, uses a vision-language model to detect and repair visual errors, and distills past runs into a growing library of reusable skills; it posts the best average score in all six agent configurations and beats a verifier-only baseline without skill evolution.
The Past Is Prologue: A Plug-in Controller for Selective Updates in Sequentially Evolving LLM Memory (Chen et al., arXiv 2026) - Proposes Janus, a method-agnostic controller that wraps existing LLM-agent memory updaters and decides whether to accept or reject each candidate update.
E-mem: Multi-agent based Episodic Context Reconstruction for LLM Agent Memory (Wang et al., arXiv 2026) - E-mem is a hierarchical multi-agent memory framework where assistant agents retain uncompressed memory segments and a master agent coordinates planning to reconstruct episodic context, evaluated on the LoCoMo benchmark. [code]
AMV-L: Lifecycle-Managed Agent Memory for Tail-Latency Control in Long-Running LLM Systems (Bamidele et al., arXiv 2026) - Proposes AMV-L, an agent-memory system that assigns each memory item a continuously updated utility score and uses value-driven promotion, demotion and eviction to keep retrieval to a bounded candidate set; against TTL retention it raises throughput 3.1x and cuts median latency 4.2x, and the gain comes from bounding retrieval work rather than from shorter prompts.
From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills (Huang et al., arXiv 2026) - Proposes a utility-grounded evaluation framework spanning experience generation, skill extraction, and skill consumption for model-generated LLM agent skills.
MemFail: Stress-Testing Failure Modes of LLM Memory Systems (Garg et al., arXiv 2026) - Introduces MemFail, a diagnostic benchmark of five datasets across four tasks that isolates and stress-tests the summarization, storage, and retrieval failure modes of external memory systems used by LLM agents.
Filesystem-Based Memory for LLM Agents: Organization, Evolution, and Sustainability (Zhou et al., arXiv 2026) - Finds that the markdown-directory memory agents actually ship buys search economy (roughly half the retrieval cost on large material) but not better answers, and its organization erodes as the store grows.
Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory (Li et al., arXiv 2026) - Shows that memory fails when the needed fact does not resemble the query: backbones answer 84.0% of indirect questions with the memory in context, six retrieval systems at most 14.4%.
Metis: Memory Foundation Model (Zhang et al., arXiv 2026) - Moves memory into the backbone rather than an external module: a native memory state maintained gradient-free by a single forward pass, with weights frozen at inference and checkpoints released.
When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents (Sun et al., arXiv 2026) - Measures what happens when a confident stored memory contradicts what the agent can see: vision F1 spans 0.887 down to 0.067 on identical grids, and an agent that trusts raw memory dies more than twice as often as the same agent given no memory at all.
Caching for the Future: Scrub Jay Episodic Memory Principles for Agent Memory Systems (Bhandari et al., arXiv 2026) - Borrows type-conditioned forgetting from scrub jay episodic memory, tagging each stored item with a perishability coefficient and a utility horizon so stale facts decay out of retrieval; ablating the decay term collapses generalization by 5.7 times.
What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics (Liu, arXiv 2026) - Compression that leaves task completion statistically flat can still roughly triple an agent's retrieval calls as it reacquires dropped state (GPT-5.5 goes from 80% to 85% completion at p = 1.0 while retrieval rises from 21.0 to 63.9 calls at p = .002), and the same sliding compression produces no retrieval surge in ALFWorld, so the signature is environment-dependent.
When Your Agent Opens the Chat App: Agent-Controlled Search over Raw Chat Logs Rivals Structured Memory (Li et al., arXiv 2026) - An agent given an iterative keyword-search loop over the unmodified chat archive, indexed lexically at turn granularity with no summaries, embeddings, trees or graphs built in advance, reaches the highest mean accuracy of any system compared (58.2 against 53.2 for HippoRAG 2) across roughly 2,800 questions under MemoryAgentBench's incremental multi-turn setting on a matched GPT-4o-mini backbone.
ForeDreamer: A Self-Evolving Dual-Agent Memory Architecture for Future Event Prediction (Zhong et al., EMNLP Findings 2026) - Converts raw web evidence into structured memory before the forecast instead of feeding retrieval results straight to the agent, separating a question-specific factual memory from experiential memory that persists across forecasting episodes, with a memory subagent building the former and two evolution tracks improving both the forecast and the memory construction; evaluated on Prophet Arena and FutureX. [code]
Corpus2Skill: Distilling Enterprise Knowledge into Navigable Agent Skills for QA and RAG (Sun et al., EMNLP Findings 2026) - An offline compiler distills the corpus into a hierarchical skill directory that the agent then navigates at serve time, drilling from a bird's-eye view down through finer summaries to documents and backing out of branches that go nowhere, rather than issuing a fresh query per question; across eleven datasets navigation turns out not to be a universal replacement, winning on five, tying on three and losing on three, with the gains concentrated in single-domain corpora whose topical taxonomy can be recovered and flat retrieval still preferable on open-domain factoid pools. [code]
Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability (Goyal et al., arXiv 2026) - Holds the stored history fixed and swaps the model, showing that memory format decides whether an upgrade is survivable: a fixed-schema knowledge graph moves accuracy by 0.0004 points, model-compressed notes swing asymmetrically by +9.91 or -13.28 points depending on migration direction, a half-migrated embedding index captures only 4.96 of the 11.90 points full re-embedding recovers, and store-only repair of notes misses a 90 per cent recovery target in all 48 histories unless the raw source was kept.
The Memory Trust Gap: Capability-Dependent Failures in Persistent-Memory Agents (Hu et al., arXiv 2026) - Separates the case where memory is the only source from the case where an authoritative tool holds the right value, and reads the failure as over-trust rather than confusion: models answer from a stale stored value 0.92 to 1.00 of the time when memory is all they have, and the harm under the trap condition is capability-gated, with the larger models in a Qwen3 size series collapsing hardest once a stale note is dressed up as current.
Dual-Layer Agentic Memory with Fast Write Routing and Slow Consolidation (Li et al., arXiv 2026) - Turns each memory write into a three-way choice (skip, write new, update) made by a 1.7B-to-8B cascade, pruning up to 68% of redundant memory while escalating under half the inputs and keeping over 98% of full-retention exact match.
Jev-Mem: System-One-Controlled Agentic Memory for Efficient AI Agents (Jiang et al., arXiv 2026) - Hands memory typing, routing, graph traversal and stopping to Jev and calls the LLM only to reason over what comes back, building LoCoMo memory 6.6x faster than the fastest baseline and scoring 0.777 against 0.700; most of that margin comes from the adversarial questions. [code]
From a Prompt to Repertoires: Evolving Functional REpertoires Enable LLM Continual Learning (Liu et al., arXiv 2026) - Adapts a frozen LLM to a sequence of tasks by keeping a repertoire of prompts where prompt optimization keeps one: an update goes into a prompt only if the prompt then keeps at least 90% of its score on the examples it already handles, is otherwise tried on the other prompts, and becomes a new prompt when none can take it, with an LLM selector choosing a prompt for each input; on a ToolUse, FinQA and SciKE-Bio sequence, with Qwen3-8B answering and GPT-5.6-Luna writing the updates, FinQA accuracy falls 1.56 points after the last task against 25.10 for GEPA and 9.43 for ACE, and in the one agent test, a minimal scaffold on six Continual Learning Bench environments, it averages 18.0 against 9.5 for GEPA with Qwen3-8B and 36.3 against 29.5 with GPT-5.6-Luna.
Reliable Memory from Unstable Growth: Curating Self-Written Agent Playbooks with Randomized Experiments (Pham et al., arXiv 2026) - Decides which lessons in an agent's self-written playbook to keep by experiment rather than by the agent's own judgment: related lessons are grouped into blocks, each block is switched on or off at random during practice tasks, a regression of success estimates each block's effect, and the curated playbook is served only if it beats the whole and the empty one on fresh practice; over 19 settings with ACE and Dynamic Cheatsheet on AppWorld, WebShop and ALFWorld it averages 66.9 against 55.5 for the whole playbook and 53.8 for the agent's own picks, though simply choosing between the whole and the empty playbook already reaches 66.4.
Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents (Xie et al., arXiv 2026) - Tests whether agent memory systems can delete what a user asks them to forget: 50 synthetic multi-session dialogues carry 135 forget requests of six types and 1,742 questions, and each has a control version without the requests, so a system earns credit for forgetting only what it had stored; across Mem0, A-MEM, LangMem, LightMem, SimpleMem, naive RAG and a full-context LLM, no system passes more than 1.5% of requests end to end, because each either leaves the information extractable under simple attacks or deletes far more than was asked. [code]
Not All Experience Belongs in the Weights: Component Routing for Self-Improving GUI Agents (Wu et al., arXiv 2026) - Splits a GUI agent's own experience into locators, procedures, state facts and lessons, and tests each one both in the weights, by fine-tuning, and in the context, as retrieved notes: locators gain 4.9 points more in the weights and lessons 1.5, a margin whose interval includes zero in half the cells, while procedures and state facts gain 4.2 and 4.1 more in the context; routing each component by a rule based on how often it recurs and whether it depends on the current state beats the better single destination by 3.5 points on average, over three 7 to 8B backbones on MobileGym and AndroidWorld.
🔧 Tool Use (51)
Corresponds to §6 (Tool Use and Action Execution).
Show 51 papers
TALM: Tool Augmented Language Models (Parisi et al., arXiv 2022) - Early, influential formulation of self-supervised bootstrapping for tool use in LMs, directly anticipating Toolformer's self-supervised approach.
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs (Li et al., EMNLP 2023) - One of the earliest and most cited dedicated benchmarks for evaluating and training tool-augmented dialogue LLMs. [code]
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (Lu et al., NeurIPS 2023) - Prominent example of natural-language-planned tool composition/orchestration across heterogeneous tool types, widely cited in tool-use and compositional-reasoning literature. [code]
Large Language Models as Tool Makers (Cai et al., arXiv 2023) - Foundational for the tool-creation (as opposed to tool-use) direction, showing LLMs can author their own reusable tools rather than only calling pre-existing ones. [code]
GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction (Yang et al., NeurIPS 2023) - Widely used reference for open-source instruction-tuning approaches to multimodal tool use, complementing proprietary-model tool-use papers. [code]
ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings (Hao et al., NeurIPS 2023) - An influential alternative architecture for scalable tool selection that avoids the context-length bottleneck of prompt-based tool listing. [code]
ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases (Tang et al., arXiv 2023) - Key evidence that generalized tool-use capability is achievable in small open models via automatically synthesized tool-use corpora, influential for later synthetic-data tool-training pipelines. [code]
RestGPT: Connecting Large Language Models with Real-World RESTful APIs (Song et al., arXiv 2023) - Demonstrates tool-use extended to complex, stateful real-world REST APIs beyond toy tool sets, with a benchmark still used for API-grounded agent evaluation. [code]
ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (Qin et al., ICLR 2024) - One of the largest and most cited tool-use datasets/frameworks, establishing ToolBench as a standard training/evaluation resource for open-source tool-use LLMs. [code]
Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models (Hsieh et al., arXiv 2023) - Reframes tool-use elicitation around documentation rather than demonstrations, an important and frequently cited methodological insight for scaling to many tools.
Small LLMs Are Weak Tool Learners: A Multi-LLM Agent (Shen et al., EMNLP 2024) - Influential for the multi-agent/role-decomposition approach to tool learning, especially relevant to efficiently deploying tool-use in smaller open models. [code]
StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models (Guo et al., ACL 2024) - Widely used evaluation infrastructure paper that fixed reproducibility problems plaguing large-scale real-API tool-learning benchmarks. [code]
What Are Tools Anyway? A Survey from the Language Model Perspective (Wang et al., COLM 2024) - A dedicated, LM-centric survey of tool use that is directly citable as a sub-topic survey reference for a broader LLM-agent survey.
ToolACE: Winning the Points of LLM Function Calling (Liu et al., arXiv 2024) - Represents the state-of-the-art synthetic-data pipeline approach to training small open models for accurate function calling, rivaling GPT-4.
xLAM: A Family of Large Action Models to Empower AI Agent Systems (Zhang et al., arXiv 2024) - Prominent industrial (Salesforce) effort establishing 'large action models' as a distinct model class optimized for agentic tool use; its five models (1B to 8x22B) topped the Berkeley Function-Calling Leaderboard at release and became a common baseline. [code]
The Berkeley Function Calling Leaderboard (BFCL): From Tool Use to Agentic Evaluation of Large Language Models (Patil et al., ICML 2025) - The de facto standard leaderboard/benchmark for comparing LLM function-calling and tool-use performance, referenced by nearly all subsequent function-calling papers. [code]
Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions (Hou et al., arXiv 2025) - Security analysis of the MCP ecosystem across the server lifecycle; the reference for protocol-layer supply-chain risk.
UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents (Liang et al., arXiv 2026) - Unifies tool-calling representation, a 22K+ tool / 390K+ instance corpus, and seven standardized benchmarks into one framework. [code]
Skill Retrieval Augmentation for Agentic AI (Su et al., arXiv 2026) - Lets agents retrieve and apply skills on demand from large libraries; introduces SRA-Bench with ~26K skills.
ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents (Soni et al., arXiv 2026) - Diagnostic benchmark separating the ways tool use fails, rather than scoring only end-task success. [code]
Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It (Hao et al., arXiv 2026) - Identifies why multi-turn tool-use RL collapses and which supervisory signals prevent it. [code]
When Does Restricting a Coding Agent to execute_code Help? A Regime × Agent-Design Ablation (Yang et al., arXiv 2026) - Ablation asking when narrowing a coding agent to a single execute_code action helps, and in which regimes it hurts. [code]
PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents (Du et al., arXiv 2026) - Co-trains multi-turn tool-use agents on privileged traces, transferring information available at training but not at inference. [code]
PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystems (Liu et al., arXiv 2026) - Long-horizon planning benchmark for tool-use agents in large tool ecosystems, where retrieval and selection dominate. [code]
MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (Bandi et al., arXiv 2026) - Large-scale tool-use benchmark built on real Model Context Protocol servers rather than synthetic tool stubs. [code]
Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions (Hasan et al., arXiv 2026) - Shows MCP tool descriptions carry recurring quality smells and that fixing them measurably improves agent tool-use.
LLM Agents Already Know When to Call Tools -- Even Without Reasoning (Sun et al., arXiv 2026) - Finds agents already encode when to call a tool without an explicit reasoning trace, questioning the need for reason-before-call. [code]
Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems (Kujanpää et al., arXiv 2026) - Replaces an LLM agent's inference-time code-generation loop with an offline agentic tool-making pipeline that compiles recurring standard-operating-procedure steps into validated, versioned tools; in a production alarm-triage system the tool calls cut p50 latency by 42%, and on 1,500 historical alarms they cut the end-to-end error rate by up to 53%.
Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents (Chen et al., arXiv 2026) - Using per-segment attention metrics on BFCL failure cases across models from 0.5B to 32B parameters, the paper localizes tool-selection errors to the decision stage rather than to retrieval.
HyperTool: Beyond Step-Wise Tool Calls for Tool-Augmented Agents (Du et al., arXiv 2026) - Folds deterministic tool subroutines into one outer call the model issues as code, so intermediate values move locally instead of through the reasoning trace; on MCP-Universe this lifts Qwen3-32B from 15.7% to 35.3%.
Tool-Aware Optimization with Entropy Guidance for Efficient Agentic Reinforcement Learning (Cao et al., arXiv 2026) - Stabilizes agentic RL by discarding rollouts whose tool calls all fail or whose outcomes are uniformly right or wrong, then adding an entropy bonus at post-tool-call tokens to keep exploration alive at the decision points that matter.
SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents (Wang et al., arXiv 2026) - Proposes SENTINEL, a failure-driven reinforcement learning pipeline in which a Controller mines failure modes from the policy's own failed rollouts, a Proposer converts them into targeted training tasks, and the Solver trains on those tasks; on Tau2-Bench Retail with Qwen3-4B it lifts pass^1 from 66.4 to 74.9.
SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models (Jiang et al., arXiv 2026) - SCRIBE is a reinforcement learning framework that grounds process-level reward modeling for tool-augmented agents in a curated library of skill prototypes.
CodeDelegator: Mitigating Context Pollution via Role Separation in Code-as-Action Agents (Fei et al., arXiv 2026) - Proposes CodeDelegator, which separates a persistent Delegator agent for strategic planning from fresh Coder agents that execute sub-tasks with clean contexts.
PruneTIR: Inference-Time Tool Call Pruning for Effective yet Efficient Tool-Integrated Reasoning (Zhang et al., arXiv 2026) - Proposes an inference-time framework that prunes erroneous tool calls in tool-integrated reasoning via Success-Triggered Pruning, Stuck-Triggered Pruning and Resampling.
The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration (Xu et al., arXiv 2026) - A survey reviewing multi-tool LLM agents, organizing recent progress across six dimensions (planning/execution, training, safety, efficiency, capability development, and evaluation) and their applications in software engineering, enterprise workflows, graphical user interfaces, and mobile systems.
AppWorld-UL: Benchmarking Diverse Agent-User Interactions for Tool-Use (Chen et al., arXiv 2026) - Claude Opus 4.7 solves only 48.6% of 516 user-in-the-loop tool tasks that require clarification, confirmation or a refusal, falling to 21.3% on compositional scenarios.
HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following (Panavas et al., arXiv 2026) - Hand an agent a 20 to 124 page policy document plus MCP tools and it stops obeying: the best of thirty configurations passes 36.2% of trials under strict grading. [code]
ToolAtlas: Learning Once, Reusing Everywhere with Tool-Side Memory (Fang et al., arXiv 2026) - Puts the memory on the provider side instead of the agent: execution-probed records of tool capabilities, failure boundaries and compositions lift pass@1 by up to 21.61% and transfer across agent frameworks without retraining.
The Bitter Lesson of Tool Calling (Patel et al., arXiv 2026) - Compares programmatic tool calling against native JSON across 14 models on BFCL v4: exposing tools as typed Python stubs the model invokes through code matches or beats JSON in 11 of 14, gains 10.6% on the GPT-5.6 family, and holds steady under context rot where the baseline slips.
Diagnosing Tool-Selection Reasoning in LLM Agents with Canary Tools (Anand et al., arXiv 2026) - Plants diagnostic decoy tools in an MCP tool set so a bare wrong-tool outcome becomes a profile of why, across six probe types and 8,640 runs; susceptibility varies about 36x across models and does not track capability tier.
The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior (Xu et al., arXiv 2026) - Holding the underlying information and actions fixed and varying only how they are exposed, six tool architectures over 11,700 repository-level issue-fixing trajectories show structured low-level interfaces raising consistency across repeated attempts by up to 4.7x and Python CodeAct-style interfaces matching task performance with 41.6% fewer steps and 56.3% lower token usage, while text-based cognitive-scaffolding tools change little.
Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning (Shao et al., arXiv 2026) - Replacing the images that crop and zoom tools return with a text placeholder matches or exceeds full thinking-with-images across LoRA, full fine-tuning and RL, so the load-bearing signal is the structured text emitted at call time rather than the returned pixels; latency falls 29 to 46% and tool-execution API calls disappear.
Can MCP Clients Decide What to Do After Failure? A Result-Only Actionability Audit (Mehan, arXiv 2026) - Asks what deterministic software can decide from a completed MCP failure result alone, and in a deliberately small sample of 21 induced failures across ten reachable servers the typed fields make failure visible in 18 cases and a broad policy readable in 8, while never exposing a specific cause, target, executable repair or replay constraint, which leaves recovery resting on prose somebody has to interpret.
One Policy Is Enough: Single-Agent Reinforcement Learning Outperforms Tree Search for Chemistry Tool Learning (Dariani et al., arXiv 2026) - Replaces a hierarchical evolutionary tree search running separate policy and execution models under two learned critics with a single left-to-right generation trained by outcome-level reinforcement against a programmatic reward read off the gold call chain, keeping no learned critic and no judge in the training loop, and still improves Tool F1 by 5.5 per cent and Return F1 by 9.6 per cent on Qwen-2.5-7B at one model invocation per question.
DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training (Gandhi et al., arXiv 2026) - Trains long-horizon tool-using agents with GRPO when no ground-truth success signal exists, by generating task-specific rubrics during training, scoring each trajectory once with an LLM judge, and redistributing the advantage over the steps responsible for each annotated criterion in closed form, with no trained attribution module; on AppWorld it gains 15.9 points over the base model and beats GRPO trained on the ground-truth reward by 5.3 despite using no verifier itself, and carries 5.3 points to Tau-Bench without a frontier judge. [code]
Do Tool Calls Execute as Intended? Measuring and Repairing Intent-Execution Correspondence in LLM Agents (Yang et al., arXiv 2026) - Checks whether the action a harness runs is the one in the tool call the LLM emitted, and names the first hop that changed it: in 47,828 shell calls from production sessions on Windows, Claude Code's Bash tool changes 12.0% of the calls that carry code, escape sequences or long text, 80.7% of calls whose backslashes are changed run the wrong action with no error, and trajectory-based judges blame the LLM for 95.1% of failures although the path from tool call to execution caused more than half; IntAct recovers 79.2% of the benchmark's failures with a changed call, while on Linux the arguments arrived as written in all 1,779 calls. [code]
ProgAgent: Learning Tool Orchestration through Programmatic Training (Li et al., arXiv 2026) - Trains Qwen3-8B and 14B to orchestrate tools by writing code that calls them and keeps intermediate results in runtime variables rather than in the context, on 215 synthesized tasks at four environment sizes with cold-start SFT and then GSPO: the 14B model reaches 29.7% on MCPMark against 8.0% for the same recipe with direct tool calls, and 45.6% on MCP-Atlas against 29.7% and Claude Sonnet-4.5's 43.8%, but 40.7% of its Playwright code calls hold a single tool invocation with no control flow, and untrained backbones gain little from the code interface alone.
Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agents (Li, arXiv 2026) - Grades 25,930 episodes of nine models, in a minimal scaffold and under Copilot CLI, Hermes and Codex CLI, against a ledger of the side effects actually committed in a sandbox of six simulated services with twelve injected faults: told to act exactly once, frontier models duplicate only 0.5% of writes whose acknowledgement was lost, but 56% under late commits and 74% under redelivery, where a read-back finds nothing and the tool contract explains 81% of the variance while the harness matters little; an idempotency key on every write cuts duplicates from 28% to 4%, and agents reported success in 90% of the episodes where they had duplicated.
Teaching LLMs to Learn Tool Trialing and Execution through Environment Interaction (Gao et al., arXiv 2026) - Trains a model to try tools out before relying on them: SFT on DeepSeek-V3.1 trajectories, built from 1,500 ToolBench queries, that test each tool and self-correct, then GRPO on 800 more. On StableToolBench, Qwen2.5-7B goes from 43.4 to 66.4 in solvable pass rate, ahead of the best RL baseline at 58.4 and of GPT-4o zero-shot at 60.2, and Qwen3-14B reaches 71.1; on the out-of-domain ToolHop the 7B model still trails GPT-4o, 37.4 against 45.3, the pass rates are graded by GPT-4o, the extra trial calls add latency, and the authors warn that trying out tools with side effects such as data writes is unsafe. Code, training data and the 7B checkpoint are released, and the released pipeline generates its training trajectories and runs RL against MirrorAPI-Cache, a model that simulates the APIs. [code]
TabAgent: Distilling LLM Agent Components into Cheap Classifiers (Levy et al., arXiv 2026) - Replaces an agent's repeated closed-set LLM calls, such as choosing which tools to shortlist, with a CatBoost classifier trained on features of the agent's own successful runs: on AppWorld it recovers 92 to 100% of the tools the GPT-5.4 shortlister picked (Recall@9) in about 3 ms instead of 7.5 s, at under 1/100,000 of the cost, and swapped into a live CUGA agent on 24 held-out tasks it reaches 27.1% task success against 29.2%, a difference whose 95% interval runs from -18.8 to +16.7 points. A fine-tuned Llama-3.1-8B still scores higher on four of the five AppWorld apps, the classifier needs a pool of successful runs (118 to reach 90% of its AppWorld score) and has to be rebuilt for a different agent scaffold, on routing and gating it only ties the best fixed choice, and the CUGA runs behind the AppWorld numbers are not released, though the other benchmarks' results can be rerun from the code. [code]
🤝 Multi-Agent Systems (57)
Corresponds to §7 (Multi-Agent Systems).
Show 57 papers
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (Li et al., NeurIPS 2023) - One of the earliest and most cited frameworks establishing autonomous agent-to-agent cooperation via role-play. [code]
Improving Factuality and Reasoning in Language Models through Multiagent Debate (Du et al., ICML 2024) - Seminal multi-agent debate paper popularizing 'society of minds'-style debate as a test-time technique. [code]
Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate (Liang et al., EMNLP 2024) - Diagnoses a core failure mode motivating why multi-agent debate helps beyond self-consistency. [code]
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate (Chan et al., ICLR 2024) - Shows multi-agent debate improves reliability of LLM-as-judge evaluation. [code]
AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation (Wu et al., arXiv 2023) - Widely adopted industrial multi-agent orchestration framework (Microsoft). [code]
ChatDev: Communicative Agents for Software Development (Qian et al., ACL 2024) - Widely cited demonstration of end-to-end multi-agent collaboration for a complex real-world workflow. [code]
AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors in Agents (Chen et al., ICLR 2024) - General-purpose, dynamically-composed multi-agent collaboration framework and study of emergent social dynamics. [code]
Building Cooperative Embodied Agents Modularly with Large Language Models (Zhang et al., ICLR 2024) - Extends multi-agent collaboration and communication to embodied/physically-grounded settings. [code]
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (Chen et al., ACL 2024) - Combines heterogeneous LLM backbones with confidence-weighted persuasion/voting in agent debate/consensus. [code]
Dynamic LLM-Agent Network: An LLM-Agent Collaboration Framework with Agent Team Optimization (v2 retitled: A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaboration) (Liu et al., arXiv 2023) - Introduces dynamic agent-team composition/topology optimization for multi-agent collaboration. [code]
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication (Yin et al., EMNLP 2023) - Provides a taxonomy of inter-agent communication paradigms, useful for surveying communication mechanisms. [code]
LLM-Deliberation: Evaluating LLMs with Interactive Multi-Agent Negotiation Games (v2 retitled: Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation) (Abdelnabi et al., arXiv 2023) - Extends multi-agent LLM research into strategic/competitive communication (negotiation). [code]
Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration (Wang et al., ACL 2024) - Boundary case showing multi-agent-style collaboration can be simulated within a single model via personas. [code]
Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs (Smit et al., ICML 2024) - Important critical/empirical counterpoint on when multi-agent debate actually helps. [code]
Debating with More Persuasive LLMs Leads to More Truthful Answers (Khan et al., ICML 2024) - Connects multi-agent debate to AI-safety scalable oversight. [code]
Mixture-of-Agents Enhances Large Language Model Capabilities (Wang et al., ICLR 2025) - Influential architecture showing structured multi-agent aggregation can outperform any single strong proprietary model. [code]
More Agents Is All You Need (Li et al., TMLR 2024) - Crucial baseline showing much of multi-agent benefit can come from ensemble scaling rather than communication. [code]
Beyond Self-Talk: A Communication-Centric Survey of LLM-Based Multi-Agent Systems (Yan et al., arXiv 2025) - Most directly on-topic recent survey for communication within multi-agent LLM systems.
Multi-Agent Collaboration Mechanisms: A Survey of LLMs (Tran et al., arXiv 2025) - Recent dedicated survey providing a structured taxonomy of collaboration mechanisms.
Language Agents as Optimizable Graphs (Zhuge et al., ICML 2024) - GPTSwarm: formalizes multi-agent systems as computational graphs whose prompts and edges are jointly learned. [code]
Scaling Large Language Model-based Multi-Agent Collaboration (Qian et al., arXiv 2024) - Scales collaboration to 1000+ agents in explicit topologies; irregular graphs outperform regular ones (a collaborative scaling result).
AFlow: Automating Agentic Workflow Generation (Zhang et al., ICLR 2025) - Searches over code-represented workflows to discover agentic pipelines automatically.
Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration (Yun et al., arXiv 2026) - Graph-based agent selection with directed/reverse message passing that outperforms Mixture-of-Agents using fewer agents. [code]
Latent Agents: A Post-Training Procedure for Internalized Multi-Agent Debate (Yi et al., arXiv 2026) - Distills multi-agent debate into a single model via two-stage fine-tuning, cutting tokens up to 93% while preserving steerable perspectives. [code]
Uno-Orchestra: Parsimonious Agent Routing via Selective Delegation (Cui et al., arXiv 2026) - Unified RL policy jointly deciding decomposition depth and worker delegation, beating workflow baselines at 10x lower cost.
Competition and Cooperation of LLM Agents in Games (Yao et al., arXiv 2026) - Finds LLM agents cooperate rather than reach Nash equilibria in resource-allocation and Cournot games, driven by fairness-based reasoning.
Multi-Agent LLMs Fail to Explore Each Other (Choi et al., arXiv 2026) - Frames peer exploration as a partially observable stochastic game and finds current agents myopic and polarized when probing each other; the fix, MACE, uses structured peer selection, and the paper proves the value of exploration rises with agent diversity. [code]
Who Broke the System? Failure Localization in LLM-Based Multi-Agent Systems (Xia et al., arXiv 2026) - Localizes which agent broke a multi-agent run, a prerequisite for debugging that aggregate success rates hide.
When is Routing Meaningful? Diversity and Robustness in Language Model Societies (Huot et al., arXiv 2026) - Asks when routing across a society of models is meaningful, and when diversity buys robustness rather than noise.
What LLM Agents Say When No One Is Watching: Social Structure and Latent Objective Emergence in Multi-Agent Debates (Ghaffarizadeh et al., arXiv 2026) - Observes what agents say with no audience, surfacing latent objectives and social structure that task metrics miss.
Decision Protocols in Multi-Agent Large Language Model Conversations (Kaesberg, arXiv 2026) - Compares decision protocols in multi-agent conversations, treating the voting or consensus rule as a design variable.
The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break (Wang et al., arXiv 2026) - Diagnoses where and why agentic systems break on long-horizon tasks, arguing much apparent long-horizon competence is a mirage.
GoAgent: Group-of-Agents Communication Topology Generation for LLM-based Multi-Agent Systems (Chen et al., arXiv 2026) - Generates the communication topology of a multi-agent system as a group-of-agents graph rather than fixing it by hand.
Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces (Zhang et al., arXiv 2026) - Trains a multi-agent system end to end from orchestration traces with reinforcement learning.
Learning Latency-Aware Orchestration for Multi-Agent Systems (Shi et al., arXiv 2026) - Targets the critical execution path rather than total cost, learning latency-aware execution graphs during training and pruning redundant agent interactions at run time, for over 50% lower end-to-end latency at competitive accuracy.
ProACT: Towards Breakdown-Aware Proactive Agent in Multi-User Collaboration (Yang et al., arXiv 2026) - Proposes ProACT, a framework that lets conversational agents observe speaker-attributed multi-user dialogue, detect whether the current turn contains a collaboration breakdown warranting intervention, and decide whether to stay silent or step in with a targeted collaboration skill; across 3,244 turn-level examples and five backbones it beats direct chat on appropriateness, non-interruptiveness and conciseness.
MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarks (Ke et al., arXiv 2026) - Formulates LLM multi-agent coordination as a reinforcement learning problem that generates an entire multi-agent system at once (holistic orchestration) and introduces MASBENCH, a controlled benchmark with five task dimensions.
Security Threat Modeling for Emerging AI-Agent Protocols: A Comparative Analysis of MCP, A2A, Agora, and ANP (Anbiaee et al., arXiv 2026) - Applies threat modeling to four AI-agent communication protocols (MCP, A2A, Agora, ANP), introducing a qualitative risk framework that identifies twelve protocol-level risks across the creation, operation and update phases, plus an MCP case study that measures how often a tool runs from the wrong provider when several servers are composed.
WebWeaver: Breaking Topology Confidentiality in LLM Multi-Agent Systems with Stealthy Context-Based Inference (Xiong et al., arXiv 2026) - Proposes WebWeaver, an attack framework that infers the communication topology of LLM multi-agent systems by compromising a single agent and reasoning solely from agent contexts rather than agent IDs.
Towards Self-Improving Error Diagnosis in Multi-Agent Systems (Li et al., arXiv 2026) - Introduces ErrorProbe, a self-improving framework for failure attribution in LLM multi-agent systems that uses backward tracing and a Strategist/Investigator/Arbiter team with verified episodic memory to localize the responsible agent and the step where the error began, outperforming baselines on TracerTraj and Who&When, most clearly at the step level.
OrchBench: Evaluating Multi-Agent Orchestration Plans in Isolation via Deterministic Simulation (Ren et al., arXiv 2026) - Scores orchestration plans by simulation instead of running the workers, tracking real execution quality at r=0.816 on 1.3% of the tokens; preserving task-critical information beats adding agents.
Two Calls Beat Five Agents: Evaluating Multi-Agent Pipelines Against Self-Refinement for Local Language Models (Prajapati et al., arXiv 2026) - Two-call self-refinement beats a five-role pipeline on a local 7B model (86.2% vs 82.0% on GSM8K, with 7.4x lower token usage), and switching JSON to plaintext matters more than architecture.
Do Latent Channels Actually Communicate? A Causal Audit of Latent Multi-Agent LLM (Zhang et al., arXiv 2026) - Replaces latent messages across examples to show aggregate accuracy hides the mechanism: a -1.00 point GSM8K effect splits into -6.17 from an unrelated message and +5.17 from example-specific content.
When Does Latent Communication Pay? A Causal Audit of Relayed KV Caches in Multi-Agent LLMs (Cheng et al., arXiv 2026) - Audits the claim that relaying KV caches between agents transfers latent thoughts, by swapping in deranged, zeroed and moment-matched caches: where the receiver needs the sender's private information the effect is real, and where it does not, the reported gain is statistically equivalent to nothing.
Everyone Conforms, No One Believes: Pluralistic Ignorance in LLM Agent Populations (YS, arXiv 2026) - Finds that agent populations reproduce pluralistic ignorance, conforming publicly 64 to 94% of the time while privately rejecting the norm, and that a single public dissenter breaks the false consensus less than 26% of the time for seven of eight models.
CityReal: Human-Aligned Urban Behavior and City Dynamics Simulation with Large-Scale LLM Agents (Bougie et al., arXiv 2026) - Large-scale human-aligned simulation of urban behavior and city dynamics; intention-driven agents learn habits and preferences via textual adapters to match real population statistics.
When Agents Coordinate: Measuring Coordination in Multi-Agent AI Coding (Destefanis et al., arXiv 2026) - Turns 1902 agent-team coding runs into temporal networks of messages, file writes and file reads: shared files replace repeated one-to-one messaging and cut output tokens by about 42% at eight agents on message-heavy work, naming one agent coordinator creates no communication hub and no reliable improvement in success, and across 244 sealed reruns agents still reach for hidden grading material in four fifths of runs.
Debate Training Reduces Reward Hacking in RLAIF (Kenton et al., arXiv 2026) - RL finetuning a Gemini 2.5 Flash-class policy through a generator versus critic debate adjudicated by a frozen, weaker Gemini 2.5 Flash Lite judge holds judge performance through training where the single-player RLAIF baseline quickly hacks it, recovering 45% of the performance gap; without player constraints, adversarial training risks defaulting to critic judge-hacking, and critique word limits (effective up to 150 words) balance the game at the cost of critic expressive clarity.
OrchMAS: Orchestrated Reasoning with Multi Collaborative Heterogeneous Scientific Expert Structured Agents (Feng et al., arXiv 2026) - Splits orchestration from execution across two tiers for scientific reasoning: an orchestrator model reads the task, builds a domain-aware pipeline and writes the role and prompt for each expert agent it instantiates, then revises that pipeline mid-run from intermediate feedback, while a separate execution model carries out the steps, which lets backbones of different capacity and cost be mixed inside one run.
At Equal Inference Cost, Multi-Agent Structure Does Not Beat a Single Frozen Agent (Dylan et al., arXiv 2026) - Fixes the total number of model calls instead of the number of environment rollouts, and the reported advantage of a Planner-Executor-Critic team over a single evolved executor disappears: 0.769 against 0.754 on ALFWorld at p = 0.80 while spending 1.8 times the evaluation calls, with leave-one-in analysis putting the realized value entirely in the executor as planner and critic evolve to empty or inert prompts, and the team trending worse on WebShop.
A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarms (Paglieri et al., arXiv 2026) - One agent in a collective of 100 proving formal conjectures found an exploit in the evaluation system, and it spread through the shared knowledge library and then peer-to-peer messages under competitive pressure, while a separate cohort audited the fraudulent proofs, alerted peers, staged boycotts and proposed validation patches; the transparent channels that carried the exploit are the same ones that made the resistance possible, which the authors read as a commons-governance problem rather than a failure inside any single agent.
Scaling Inference-Time Compute with Teams of Coding Agents (Samway et al., arXiv 2026) - Treats coding-agent team design as compute allocation, varying the stage a lead delegates, the worker models and the team size on 200 ProgramBench tasks with GPT-5.6 Sol, Terra and Luna: teams beat solo agents but cost three to seven times as much, delegating only planning matches the reward of delegating all three stages at half the cost, cheap Luna planners under a Sol lead recover 47.5% of an all-Sol team's gain for 13.9% of its added cost while cheap implementers lower reward, and peer messaging does not raise reward; on Commit0-Lite, where the skeleton already splits the work, implementation is the stage worth delegating.
Worse Together: How Performance Breaks Down in Multi-User Multi-Agent Teams (Paliskara et al., arXiv 2026) - Compares one coordinator agent serving every user with a team in which each user has an agent, across five frontier models and 77 scenarios in four environments with shared resources, and the team does worse in every environment: on a shared API budget, peer-to-peer Claude Opus 5 teams reach 30% of the optimum against 64% for the coordinator and 7% with no channel between them, and as teams grow from four to sixteen agents the share of Sonnet 5 agents that ever act falls from 66% to 10%; mitigations such as explicit instructions help, but only in the environment they were written for.
Scaling Discovery through Test-Time Communication (Park et al., arXiv 2026) - Runs identical Copilot CLI agents that share findings through a common workspace, and compares a team of k with the best of k independent agents at the same per-agent budget: on the 25 public ARC-AGI-3 games with Claude Sonnet 4.6, a team of three matches the best of 13 and a team of five the best of 33, and a GPT-5.6 Sol team of four compresses an MNIST classifier to 1,957 bytes against 3,160 for the best of four and 2,461 for the best human solution; the gains need verifier feedback and enough compute, since teams trail at low budgets and on Terminal-Bench 2.0, 60.67% against 62.36%. [code]
Voting or Consensus? Decision-Making in Multi-Agent Debate (Kaesberg et al., ACL Findings 2025) - Changes only the decision protocol in debates among three Llama 3 8B agents, comparing four voting and three consensus rules on three knowledge and three reasoning tasks: voting beats consensus by 13.2 points on average on the reasoning tasks, mostly on MuSR and SQuAD 2.0 while StrategyQA moves 0.2, consensus wins by 2.8 on the knowledge tasks and needs 1.42 rounds on average against 3.38 for voting, approval voting reaches no decision in 59% of cases, and on StrategyQA more agents help while more rounds before the vote hurt; the proposed Collective Improvement, in which agents see only each other's last solutions, adds 7.4 points there, though every result comes from dataset subsets and debates cost five to ten times the compute of one chain-of-thought answer. [code]
Stay Focused: Problem Drift in Multi-Agent Debate (Becker et al., EACL Findings 2026) - Scores the answer after each of seven debate turns among three Llama-3.1-70B agents on ten tasks and counts a debate as drifting when its answer gets worse than at an earlier turn: 74.6% to 88.6% of debates drift on the three generative tasks against 6.6% to 15.5% on knowledge and reasoning tasks, 8.5% to 45.4% of drifting debates recover, and eight experts reading 170 drifting excerpts most often mark lack of progress (60); the DRIFTJudge detector, which needs no gold label, reaches 0.92 specificity but 0.15 precision on MMLU-Pro, and DRIFTPolicy, a feedback agent called in when DRIFTJudge flags a turn, cuts the seven-turn accuracy loss there from 3.66 to 0.08 points for Qwen2-7B agents that vote, but not under consensus, where Llama-3.1-70B loses 4.08 points with it against 2.23 without. [code]
MALLM: Multi-Agent Large Language Models Framework (Becker et al., EMNLP Demo 2025) - Open-source framework for multi-agent debate on text tasks that sets agent personas (none, expert, Big Five), response style (simple, critical, reasoning), discussion paradigm (memory, relay, report, debate) and decision protocol (voting, consensus or a judge) independently in one config file, 144 combinations in its public demo, runs on any OpenAI-compatible endpoint, loads any text dataset from Hugging Face and repeats runs to report standard deviations; Voting or Consensus? and Stay Focused were run on it, and the results in the paper are restated from the authors' earlier studies, such as the fully visible memory paradigm agreeing in 1.75 turns on StrategyQA against 2.61 for relay. [code]
🌍 Part II: Agents in Context
🌐 Interactive Environments (60)
Corresponds to §8 (Agents in Interactive Environments).
Show 60 papers
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (Ahn et al., CoRL 2022) - Foundational demonstration of LLM-as-planner grounded by real-world affordances for embodied robotic agents. [code]
Inner Monologue: Embodied Reasoning through Planning with Language Models (Huang et al., CoRL 2022) - Established the closed-loop, feedback-grounded planning pattern underlying subsequent embodied/GUI agent architectures.
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (Shridhar et al., ICLR 2021) - Widely used benchmark for evaluating LLM-based embodied/household agents (ReAct, Reflexion), bridging text reasoning and embodied execution. [code]
RT-1: Robotics Transformer for Real-World Control at Scale (Brohan et al., RSS 2023) - Foundational large-scale robot-transformer model establishing the recipe later extended by RT-2 and OpenVLA. [code]
PaLM-E: An Embodied Multimodal Language Model (Driess et al., ICML 2023) - Seminal embodied multimodal LLM showing internet-scale vision-language pretraining transfers to embodied robotic reasoning, inspiring RT-2 and VLA models.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (Brohan et al., CoRL 2023) - Established the vision-language-action (VLA) modeling paradigm central to embodied/robotics agent research.
OpenVLA: An Open-Source Vision-Language-Action Model (Kim et al., CoRL 2024) - Open-source counterpart to closed VLA models, democratizing research on LLM-driven robotic control. [code]
WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents (Yao et al., NeurIPS 2022) - Foundational, widely-used benchmark for grounded language web agents, predating and motivating later LLM-based web-navigation research. [code]
Mind2Web: Towards a Generalist Agent for the Web (Deng et al., NeurIPS 2023) - First benchmark and LLM-based agent explicitly designed for generalist web navigation on real websites; standard reference cited by subsequent web/GUI agent papers. [code]
A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis (Gur et al., ICLR 2024) - Splits an instruction into canonical sub-instructions, summarizes long HTML into task-relevant snippets, and acts through generated Python, lifting success on real websites by over 50% and topping Mind2Web offline planning.
GPT-4V(ision) is a Generalist Web Agent, if Grounded (Zheng et al., ICML 2024) - First systematic demonstration that multimodal LLMs can act as generalist visual web agents, catalyzing the shift to vision-grounded web/GUI agents. [code]
WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (He et al., ACL 2024) - Key demonstration and benchmark for real-world, multimodal browser agents widely used to evaluate subsequent web-agent systems. [code]
Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation (Chae et al., ICLR 2025) - Confirms current LLMs carry no world model, then trains one to describe in free-form text what each action would change, so the agent can simulate an outcome before committing to something irreversible like a non-refundable booking; improves policy selection on WebArena and Mind2Web more cheaply than tree search. [code]
CogAgent: A Visual Language Model for GUI Agents (Hong et al., arXiv 2023) - One of the first large VLMs purpose-built for screenshot-only GUI grounding, establishing the high-resolution visual GUI agent architecture line. [code]
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents (Cheng et al., ACL 2024) - Established GUI grounding as a core sub-problem for visual GUI agents and introduced ScreenSpot, a standard grounding benchmark. [code]
Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception (Wang et al., arXiv 2024) - Representative vision-centric mobile GUI agent design demonstrating cross-app, metadata-free operation. [code]
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (Xie et al., NeurIPS 2024) - Standard benchmark for evaluating 'computer-use' agents, used to evaluate essentially every major computer-use agent since 2024. [code]
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents (Rawles et al., ICLR 2025) - Dominant reproducible benchmark for mobile GUI agents enabling dynamic task variation. [code]
UI-TARS: Pioneering Automated GUI Interaction with Native Agents (Qin et al., arXiv 2025) - State-of-the-art open 'native' GUI/computer-use agent model showing the field's shift toward end-to-end trained GUI action models. [code]
GUI Agents: A Survey (Nguyen et al., ACL 2025) - Direct, up-to-date survey specifically on GUI agents for structuring the GUI/computer-use agent sub-area and taxonomy.
Large Language Model-Brained GUI Agents: A Survey (Zhang et al., arXiv 2024) - Complementary GUI-agent-specific survey valuable for comprehensive coverage of the LLM-driven GUI agent literature. [code]
A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Models (Ning et al., KDD 2025) - Dedicated survey for the web-agent sub-area, giving a ready-made taxonomy and trustworthiness discussion specific to browser/web automation agents.
UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning (Wang et al., arXiv 2025) - Successor to UI-TARS; multi-turn RL for end-to-end GUI control.
π0: A Vision-Language-Action Flow Model for General Robot Control (Black et al., arXiv 2024) - Flow-matching action expert on a pretrained VLM, controlling many robot embodiments. [code]
Tongyi DeepResearch Technical Report (Tongyi DeepResearch Team, arXiv 2025) - Open end-to-end deep-research agent model for long-horizon web research and synthesis. [code]
Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents (Xu et al., arXiv 2026) - GUI-Owl-1.5 native multi-platform (mobile/desktop/browser) agent family with a data flywheel and MRPO RL, SOTA on 20+ GUI benchmarks. [code]
EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience (Xue et al., arXiv 2026) - Self-evolving computer-use agent fusing synthetic-task generation with online sandbox policy optimization, reaching 56.7% on OSWorld.
CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents (Jian et al., arXiv 2026) - Releases VideoCUA, the UI-Vision benchmark, and GroundCUA (56K screenshots, 3.6M UI annotations) for computer-use agents. [code]
Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading (Li et al., arXiv 2026) - Terminal benchmark built for long-horizon tasks, where agents fail from state tracking rather than single-step incompetence. [code]
WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation (Dong et al., arXiv 2026) - Large-scale web-agent benchmark designed for efficient evaluation rather than a handful of hand-built sites. [code]
CLI-Anything: Towards Agent-Native Computer Use (Yang et al., arXiv 2026) - Argues computer use should be agent-native through the CLI instead of pixel-level screen imitation. [code]
PhoneBuddy: Training Open Models for Agentic Phone Use (Tang et al., arXiv 2026) - Trains open models for agentic phone use, a setting dominated by closed systems. [code]
Designing Agent-Ready Websites for AI Web Agents: A Framework for Machine Readability, Actionability, and Decision Reliability (Elnaffar et al., arXiv 2026) - Turns the problem around and asks how websites should be built to be machine-readable and actionable for agents.
TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks (Chu et al., arXiv 2026) - Benchmarks agents on real-world terminal tasks, where long-horizon state tracking dominates single-step skill. [code]
WebNavigator: Global Web Navigation via Interaction Graph Retrieval (Zhang et al., arXiv 2026) - Web navigation through retrieval over an interaction graph, giving the agent global structure instead of local page views. [code]
MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research (Wu et al., arXiv 2026) - Verifiable, highly parallel simulation platform for training and evaluating mobile GUI agents.
ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory (Tian et al., arXiv 2026) - Proposes an agent operating system for embodied robots that unifies planning, skill execution, and a Universal Multi-modal Graph Memory integrating visual, spatial, and temporal information. [code]
EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments (Zhilin Wang et al., arXiv 2026) - Introduces Autonomous Policy Evolution as an evaluation setting and the EvoPolicyGym benchmark, in which agent-models iteratively modify executable policy code across 16 compact interactive reinforcement learning environments. [code]
ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance (Jin et al., arXiv 2026) - Introduces ChainSWE, a benchmark of 304 chronologically chained issues across 54 Python projects that evaluates coding agents on sequential, dependent bug fixes rather than isolated defects.
VisCritic: Visual State Comparison as Process Reward for GUI Agents (Qian et al., arXiv 2026) - Proposes VisCritic, a visual process reward framework that verifies GUI agent actions by comparing pre- and post-action screenshots in visual feature space via a Siamese vision transformer and an action-aware critic head.
ASPIRE: Agentic /Skills Discovery for Robotics (Lu et al., arXiv 2026) - Proposes ASPIRE, a continual learning system in which an agent autonomously writes and refines robot control programs under a code-as-policy paradigm.
GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents (Zhou et al., arXiv 2026) - Using a matched benchmark to compare computer-use agents across GUI and CLI modalities, the paper reports that the strongest GUI agent attains a 59.1% full pass rate versus 48.2% for the strongest original-skill CLI agent.
A History-Aware Visually Grounded Critic for Computer Use Agents (Lee et al., arXiv 2026) - Proposes HiViG, a multimodal critic trained on GUI trajectories that compresses a computer-use agent's interaction history into multi-step goals and verifies proposed actions against the current screenshot. [code]
ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories (Luo et al., arXiv 2026) - Proposes ISE, a three-stage Intent-Simulate-Execute pipeline that synthesizes multi-turn OS-agent training trajectories by pairing a role-locked user simulator with real tool execution in isolated OS workspaces. [code]
Demonstration-Free Robotic Control via LLM Agents (Tsui et al., arXiv 2026) - Introduces FAEA (Frontier Agent as Embodied Agent), applying an unmodified general-purpose LLM agent framework to robotic manipulation without demonstrations or fine-tuning. [code]
On Data Engineering for Scaling LLM Terminal Capabilities (Pi et al., arXiv 2026) - Introduces Terminal-Task-Gen, a synthetic task generation pipeline, and studies data strategies (filtering, curriculum) to train Nemotron-Terminal models.
Generalization in Online Reinforcement Learning for Mobile Agents (Gu et al., arXiv 2026) - Introduces AndroidWorld-Generalization, a benchmark and GRPO-based online RL training system for GUI mobile agents, reporting zero-shot generalization gains of 26.1% on unseen instances that diminish to 15.7% on unseen templates. [code]
WebXSkill: Skill Learning for Autonomous Web Agents (Wang et al., arXiv 2026) - WebXSkill is a skill-learning framework for web agents that combines parameterized action programs with natural language guidance, extracting reusable action patterns from synthetic trajectories and organizing them in a URL-based graph for context-aware retrieval; it improves results on WebArena, WebVoyager and Online-Mind2Web. [code]
Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents (Yu et al., arXiv 2026) - First benchmark for parallel GUI agents: splitting long-horizon desktop tasks across concurrent workers on separate machines beats the best serial baseline by 12.9 points at roughly half the steps and tokens. [code]
OpenForgeRL: Train Harness-native Agents in Any Environment (Yu et al., arXiv 2026) - Trains agents end-to-end inside the real inference harnesses they are deployed with (Claude Code, Codex, OpenClaw), and finds that some harnesses are substantially harder to learn than others.
StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents (Yang et al., arXiv 2026) - Grounding computer-use agents in program state rather than screenshots lifts Claude Opus 4.8 from 20.6% to 26.9% on OSWorld 2.0 at about nine times lower cost per task.
StepReflect: Structured UI Transition Reflection for Mobile GUI Agents (Guo et al., arXiv 2026) - Treats per-step GUI reflection as structured prediction over explicit transition specifications rather than open-ended multimodal reasoning; an 8B model reaches 82.16% transition accuracy on AndroidWorld, 11.83 points above zero-shot GPT-5.2 on the same input.
ComponentBench: Diagnosing Component-Level Failures in Computer-Use Agents (Guan et al., arXiv 2026) - Benchmarks computer-use agents at the component level with 2,910 programmatically verified tasks over an ontology of 97 canonical UI components, and finds that within a single shared harness changing only the observation and action space shifts task success by more than 30% for the same model, with GPT-5 mini falling from 83.1% on accessibility-tree observations to 48.9% under coordinate-only pixel control. [code]
Neurosymbolic Embodied Agents (Albinhassan et al., arXiv 2026) - Factors household tasks into task-directed visual exploration and PDDL-constrained decoding with Monte Carlo tree search, letting open 4B to 27B models exceed 90% success on both VirtualHome and ALFWorld, where constraints or search alone solve under a third of ALFWorld tasks but their combination solves over 95%, and residual failures localize to state acquisition rather than plan generation.
CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents (Shi et al., arXiv 2026) - Builds environments where the same application state is reachable through both the screen and the command line, adapting 16 real desktop applications into reproducible virtual machines with command surfaces that are discovered, wrapped or generated, and training on the harvested trajectories moves a 9B model from inefficient clicking and brittle scripting toward using whichever interface is cheaper: 16.8 points of success on OSWorld with 57 per cent fewer steps and 44 per cent fewer tokens.
Discriminative World Models for Web Agents (Li et al., arXiv 2026) - Argues that a world model trained by supervised next-state prediction is optimized for the wrong thing once a ranker consumes it, since ranking needs predicted states that separate the candidate actions rather than states that merely look plausible, and trains instead on predicted-state matching over branching WebArena trajectories where every decision point carries the alternative actions alongside the states each of them produces.
Routing Is Least Learnable Where It Is Most Valuable: Bounds on Representation Routing for Web Agents (Wei et al., arXiv 2026) - None of five routing policies over six web-agent observation modes, a confidence cascade among them, robustly beats one well-chosen fixed mode, because a router gets labels only as often as the agent succeeds; rerunning the same mode already changes 12-14% of outcomes.
AutoEnvScaling: Automating the Data Flywheel with Terminal Agents (Yu et al., arXiv 2026) - Treats building RL environments as a terminal task for an agent: a proposer reads the solver's rollouts, then builds and tests complete tasks in a sandbox and calibrates them against the solver, so new environments aim at what the solver still fails; Qwen3.5-35B-A3B trained on environments from a GPT-5.6-sol proposer reaches 38.9 on Terminal-Bench 2.1 against 34.6 for the matched Tmax baseline, and with Qwen3.6-35B-A3B as both proposer and solver the score rises from 41.1 to 53.3, but Terminal-Bench 4.0 only moves from 1.6 to 3.2, and without the solver's rollouts in the proposer's workspace the trained solver loses 3.6 and 6.7 points with the two frontier proposers. [code]
SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire (Lowe and Keemink, arXiv 2026) - Has 20 LLMs play complete runs of the deck-builder Slay the Spire at its hardest difficulty, through tool calls, on ten held-out seeds and under a reskin that hides the game's names, and compares them with 206,082 runs from the game's telemetry and four expert streamers: Claude Fable 5.1 clears 1.60 acts per run against 0.73 for the average human run and 2.5 to 3.1 for the streamers, with the gap opening after the first act, and no method of carrying memory between runs improves play, since 17% of the stored memories are false; ten seeds are too few to order neighbouring models.
Same Tasks, Different Apps: Why Mobile GUI Agents Fail to Generalize? (Tran et al., EMNLP Findings 2026) - Keeps the user's goal fixed and changes the Android app that serves it, over 520 task and app pairs on 52 apps: 11 instruction-only agents succeed on 38.7% of tasks in the original AndroidWorld apps and on 27.3% in new ones, Gemini-3.1-Pro falls from 81.1 to 52.1 on the inherited templates, and app-independent sub-goals raise success from 37.8% to 40.9% in five categories while widening the gap there from 14.9 to 18.8 points, and lower it from 18.3% to 14.8% in the other five.
🚀 Applications (60)
Corresponds to §10 (Applications).
Show 60 papers
AutoCodeRover: Autonomous Program Improvement (Zhang et al., arXiv 2024) - One of the first cost-efficient autonomous program-repair agents grounded in structured code search. [code]
Agentless: Demystifying LLM-based Software Engineering Agents (Xia et al., arXiv 2024) - Influential counter-narrative showing simpler non-agentic pipelines can rival complex agents. [code]
OpenHands: An Open Platform for AI Software Developers as Generalist Agents (Wang et al., ICLR 2025) - The leading open community platform underlying much subsequent applied coding-agent research. [code]
Large Language Model-Based Agents for Software Engineering: A Survey (Liu et al., arXiv 2024) - A dedicated sub-topic survey giving the taxonomy needed to situate coding/SWE agent work. [code]
Autonomous chemical research with large language models (Boiko et al., Nature 2023) - One of the earliest and most-cited demonstrations of an LLM agent performing autonomous physical-world scientific experimentation. [code]
ChemCrow: Augmenting large-language models with chemistry tools (Bran et al., Nature 2023) - A foundational tool-augmented LLM agent paper for chemistry. [code]
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery (Lu et al., arXiv 2024) - A landmark, widely publicized attempt at fully automating the scientific-paper lifecycle. [code]
The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search (Yamada et al., arXiv 2025) - Marks a concrete, verifiable milestone for autonomous scientific discovery agents. [code]
Towards an AI co-scientist (Gottweis et al., arXiv 2025) - A major industry (Google) applied-agent system for hypothesis generation in science.
AlphaEvolve: A coding agent for scientific and algorithmic discovery (Novikov et al., arXiv 2025) - A landmark demonstration of an LLM agent making a genuine, verifiable new mathematical/algorithmic discovery.
Kosmos: An AI Scientist for Autonomous Discovery (Mitchener et al., arXiv 2025) - One of the most capable and rigorously evaluated 'AI scientist' agents to date.
ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery (Chen et al., ICLR 2025) - A rigorous, expert-validated benchmark quantifying the gap between current LLM agents and end-to-end scientific-discovery automation. [code]
Towards Scientific Intelligence: A Survey of LLM-based Scientific Agents (Ren et al., arXiv 2025) - The dedicated sub-topic survey for grounding the scientific-discovery-agent literature.
A Survey of LLM-based Agents in Medicine: How far are we from Baymax? (Wang et al., ACL 2025) - The primary dedicated survey of LLM agents in healthcare.
MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making (Kim et al., NeurIPS 2024) - A widely cited example of adaptive multi-agent orchestration tailored to medical reasoning complexity. [code]
Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents (Li et al., arXiv 2024) - A distinctive applied-agent paradigm for learning medical expertise via agent-agent simulation.
Towards Conversational Diagnostic AI (Tu et al., arXiv 2024) - A landmark, rigorously evaluated Google system showing an LLM agent matching or exceeding physicians in simulated diagnostic dialogue.
Large Language Model Agent in Financial Trading: A Survey (Ding et al., arXiv 2024) - The dedicated sub-topic survey needed to anchor the finance branch of applied LLM agents.
FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design (Yu et al., AAAI 2023) - An early, widely referenced LLM trading agent introducing human-cognition-inspired layered memory design. [code]
TradingAgents: Multi-Agents LLM Financial Trading Framework (Xiao et al., arXiv 2024) - A recent, popular multi-role multi-agent finance system widely used as a reference architecture. [code]
FinGPT: Open-Source Financial Large Language Models (Yang et al., IJCAI 2023) - One of the most cited open-source financial LLM/agent efforts, providing base-model infrastructure underlying many downstream financial agent systems. [code]
SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? (Miserendino et al., arXiv 2025) - Prices coding-agent competence in real freelance dollars; frontier models leave most of the posted value unearned. [code]
AgentClinic: A Multimodal Agent Benchmark to Evaluate AI in Simulated Clinical Environments (Schmidgall et al., arXiv 2024) - Multimodal benchmark of simulated clinical settings for doctor-patient agent interaction.
OptimAI: Optimization from Natural Language Using LLM-Powered AI Agents (Thind et al., arXiv 2025) - Turns natural-language optimization problems into executable solver code through a formulator-planner-coder-critic pipeline, with UCB-based plan selection.
AutoSci: A Memory-Centric Agentic System for the Full Scientific Research Lifecycle (Qian et al., arXiv 2026) - Memory-centric agentic system automating the full scientific research loop. [code]
SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents (Wang et al., arXiv 2026) - Self-adaptive context pruning that keeps coding agents effective under long repository contexts. [code]
LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent (Li et al., arXiv 2026) - Scalable agentic-RL training framework for deep-research agents. [code]
LawThinker: A Deep Research Legal Agent in Dynamic Environments (Yang et al., arXiv 2026) - Deep-research legal agent operating in dynamic legal environments. [code]
Agentic Trading: When LLM Agents Meet Financial Markets (Xia et al., arXiv 2026) - Studies LLM agents acting in financial markets and the trading dynamics they produce.
Rethinking Scientific Discovery in the Agentic Era (Zheng et al., arXiv 2026) - Position paper on what changes, and what does not, when scientific discovery is run by agents.
Deep Research in Physical Sciences: A Multi-Agent Framework and Comprehensive Benchmark (Jiang et al., arXiv 2026) - Multi-agent deep-research framework and benchmark for the physical sciences. [code]
HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents (Liu et al., arXiv 2026) - Benchmark suite of realistic agentic healthcare environments rather than static clinical question answering. [code]
EvoDS: Self-Evolving Autonomous Data Science Agent with Skill Learning and Context Management (Yang et al., arXiv 2026) - Self-evolving data science agent combining skill learning with context management. [code]
MetaResearcher: Scaling Deep Research via Self-Reflective Reinforcement Learning in Adversarial Virtual Environments (Yu et al., arXiv 2026) - Trains the deep-research loop with self-reflective reinforcement learning against adversarial conditions.
Can Deep Research Agents Retrieve and Organize? Evaluating the Synthesis Gap with Expert Taxonomies (Zhang et al., arXiv 2026) - Evaluates the synthesis gap in deep-research agents: whether they can organize retrieved evidence, not just retrieve it, against expert references. [code]
ClinicalAgents: Multi-Agent Orchestration for Clinical Decision Making with Dual-Memory (Ge et al., arXiv 2026) - Multi-agent orchestration for clinical decision making with a dual-memory design. [code]
SciResearcher: Scaling Deep Research Agents for Frontier Scientific Reasoning (Zheng et al., arXiv 2026) - Scales deep-research agents toward frontier scientific reasoning tasks.
Physics-Audited Agentic Discovery in Scientific Machine Learning (Abueidda et al., arXiv 2026) - Selects agent-discovered surrogate models by machine-checkable physics requirements rather than error alone, catching a case where an error-matched baseline responds to future parts of the loading history and fails a causality check.
LLMoxie: Exploring Agentic AI for Scientific Software Development (Setiawan et al., arXiv 2026) - Reports twenty months of experience deploying LLMoxie, an institutional three-tiered agentic AI platform with a LiteLLM/MLflow governance control plane and an open-source Plugin-Agent-Skill ecosystem.
Agon: An Autonomous Large-Scale Omnidisciplinary Research System Built on Prompt Economy (Sun et al., arXiv 2026) - A research orchestrator that validates what can be checked inside the workflow and leaves the rest to human scientists; 444 loop iterations across disciplines with no human-written experimental code, and a failure taxonomy separating what the loop can fix from what it cannot. [code]
Hybrid-Gym: Training Coding Agents to Generalize Across Tasks (Xie et al., arXiv 2026) - Trains coding agents on synthetic auxiliary tasks such as function localization and dependency search, which transfer to real work: +25.4% on SWE-Bench Verified, +7.9% on SWT-Bench Verified, +5.1% on Commit-0 Lite. [code]
Toward Expert Investment Teams: A Multi-Agent LLM System with Fine-Grained Trading Tasks (Miyazaki et al., arXiv 2026) - Proposes a multi-agent LLM framework that decomposes investment analysis into fine-grained trading subtasks, evaluated on Japanese stock data to improve risk-adjusted returns over abstract-instruction baselines.
MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome (Ye et al., arXiv 2026) - MiroEval is a benchmark of 100 tasks (70 text-only, 30 multimodal) that evaluates deep research agents along synthesis quality, factuality, and research-process dimensions. [code]
HeartAgent: An Autonomous Agent System for Explainable Differential Diagnosis in Cardiology (Zhou et al., arXiv 2026) - HeartAgent is an autonomous multi-agent system that integrates customized tools and curated data resources and orchestrates specialized sub-agents to perform explainable differential diagnosis in cardiology.
AutoNumerics: An Autonomous, PDE-Agnostic Multi-Agent Pipeline for Scientific Computing (Du et al., arXiv 2026) - Builds and residual-verifies classical numerical PDE solvers straight from natural-language problem statements, keeping the solver transparent rather than neural. [code]
Stress-testing large language model agents in a robotic chemistry laboratory (Guo et al., arXiv 2026) - Runs 4,608 trials on a 45-workstation robotic chemistry lab: only 3.3% of agent workflows were expert-judged executable, the best system reached 28.1%, and feedback never triggered replanning.
PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents (Vatanparvar et al., arXiv 2026) - Benchmarks patient-facing health agents over 1,200 tool-using conversations; triage separates models most sharply (32% to 88% pass rates), and even the strongest scores only 4.25 of 5 overall. [code]
Agentic Evaluation of Copyright Law Compliance (Hui et al., arXiv 2026) - Copyright-Bench puts agents on commercial jobs (websites, merchandise, pitch decks) and finds they pick copyrighted works when public-domain alternatives exist, with open-weight violation rates rising under simulated time pressure.
From Social Coding to Agentic Coding: Productivity and Relational Reconfiguration in Open-Source Communities (Zhou et al., arXiv 2026) - Simulates a community of 1,084 real GitHub developers with and without coding agents: completed tasks rise 39.0% and median completion time falls from 45 to 20 minutes, while direct human-to-human interaction drops from 32.4% to 11.6% and the gains concentrate among the already well connected.
Vero: Can AI Agents Build Formally Verified Software Repositories? (Ye et al., arXiv 2026) - Benchmarks joint implementation and machine-checked proof synthesis at the repository level over 43 multi-module instances drawn from real repositories spanning Python, Dafny, Verus and Coq; the strongest frontier coding-agent configuration fully solves only 27 of 43 and closes no specifications on the hardest repositories. [code]
Auditing Self-Evolution in Financial Agents: Capability Gains, Security Drift, and Execution-Interface Mismatch (Li et al., arXiv 2026) - Audits three self-evolving agent designs (SkillOpt, Agent Workflow Memory, ReasoningBank) in simulated e-banking and finds capability and exposure rising together: SkillOpt lifts benign utility from 0.741 to 0.837 while exposure to injected content rises from 0.820 to 0.943, overall attack success rises from 0.496 to 0.530, and unauthorized financial state changes rise to 0.685.
SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents (He et al., arXiv 2026) - Derives review constraints from real pull-request review comments and scores them separately from functional correctness across 303 repository-level instances in 75 Python projects, finding that 221 of 644 patches that pass the functional tests still violate a constraint a reviewer had stated, so functional-only scoring overstates what an agent has actually delivered. [code]
Why Better Models Can Create Riskier Systems: Evidence from LLM Agents in Financial Markets (Ross et al., arXiv 2026) - Argues that shared training and architecture make more capable models behave more alike, so their correlated actions leave a risk floor no amount of diversification removes, and tests it with LLM traders in an agent-based market: correlation rises with capability, more agents lower market-level risk while the shared reasoning is accurate, and the same correlation turns into a liability once the agents share a misinformation environment.
Training Agents to Evolve with Their Harness: TaoLive Digital Avatar Agent Technical Report (TaoLive AIGC LLM Team, arXiv 2026) - Takes the position that a compact model has to be trained for a harness that keeps changing, since large models adapt to edited Skills, hooks, prompts and tool schemas without retraining but miss the latency budget a live stream imposes, and augments the harness state during training so the model never sees a fixed configuration: 94.6 on harness-variant questions against a base of 75.4, and where fixed-harness supervised finetuning costs 7.7 points of IFEval against the base model the augmented recipe gives none of it back, at P50 and P95 latencies of 3.4 and 8.1 seconds on one H20 and a positive A/B test in Taobao Live.
Catching Developers in the Flow: Low-Latency Agentic Program Repair at Google Scale (Ziftci et al., ASE 2026) - Puts a ReAct generate-and-validate repair agent on pre-submit test failures inside Google's internal developer tools, capped at 100 tool calls and 30 minutes and gated by abstention filters before and after each run; with an internal Gemini 2.5 Pro its fixes were judged correct on 67.18% of 195 failures, and in deployment it suggested fixes on 295,508 changes, of which developers applied 28,554, taking a median 9.85 minutes per successful fix against 27.48 minutes before a developer's next edit, though it sometimes proposes reverting the change or commenting out the failing tests.
From Benchmark to Bench: Can Agents Survive Real-World Drug Discovery? (Llompart et al., arXiv 2026) - Lets Claude Sonnet 4.6 run lead optimization through 24 MCP tools, writing and revising the project's objectives and either proposing molecules itself or handing generation to REINVENT 4, on nine past campaigns from GSK, Sanofi and AstraZeneca replayed with fixed temporal cutoffs: three of four GSK series and the AstraZeneca campaign met their objective, only two of four Sanofi projects did, four chemists could not tell the proposals from held-out compounds, and success followed the accuracy of the scoring models rather than the generation route; every endpoint is a prediction, not a measurement.
Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agents (Drozdov et al., arXiv 2026) - Grades legal research agents without partial credit on 413 U.S. questions written by experts, so an answer counts only if it meets every rubric criterion and the authorities it cites check out; across thirteen frontier models with web search, case-law search and retrieval tools, the best, Claude Opus 4.8, is fully correct on 42.9% while weighted pass rates run from 68% to 86%, Kimi K2.6 averages 112 tool calls for 15.5%, and removing the tools costs 22.5 points on average, with one run per question scored by an LLM judge. [code]
From Benchmarks to Business Impact: Deploying IBM Generalist Agent in Enterprise Production (Shlomov et al., IAAI 2026) - Reports IBM's pilot of its open-source CUGA agent, a planner-executor system at 61.7% on WebArena and 48.2% scenario completion on AppWorld's challenge split, in a talent-acquisition outsourcing business, where it answers recruiters' questions through 13 read-only analytics APIs with web browsing turned off: it reaches 87% accuracy on the 26 tasks of the new BPO-TA set, raises the valid first-try rate from 62% for a plain ReAct agent to 79%, and drops 11 points without reflective retries; the agent is not yet in full production, the reported savings (answers in 2 to 5 minutes instead of about 20, up to 90% less development time) are estimates from controlled tests and limited analyst feedback with no significance testing, and the paper lists BPO-TA's 26 tasks, but the APIs and data behind them are not in the code release. [code]
AgonAlpha: Autonomous Alpha Discovery via Prompt Economy and Scalable Agentic Search (Ye et al., arXiv 2026) - Mines formulaic trading alphas on WorldQuant BRAIN with two roles: a proposer that saves each candidate as a frozen report with its hypothesis, expression and platform evidence, and a reviewer in a fresh context that can rerun the evaluation and zero the candidate's score, while a scheduler that counts jobs still running splits the budget across lineages. Five of the authors ran it on separate accounts with six model backends: 17 of 60 submissions got BRAIN's top grade, the best alpha reached Fitness 9.50 and Sharpe 3.48, and the reviewer zeroed 2 of the 24 reports it audited. Every figure is BRAIN's score on the 2019 to 2023 window the formulas were also selected on; out-of-sample results are not reported, there is no baseline or ablation, and the code repository holds the prompts and the scheduler but not the run records. [code]
AutoSR: Automatic Symbolic Regression by Searching Research States (Zhang et al., arXiv 2026) - Runs symbolic regression as a search over research states, each a candidate equation kept with its motivation, numerical probes, failed attempts and an independent review, which a proposer and a reviewer agent extend under Monte Carlo tree search, alongside a loop that runs PySR, SISSO and FEX. On nine problems picked for difficulty it finds an equation algebraically equivalent to the ground truth every time, including three cp3-bench problems that none of 12 published systems solved and six LSR-Transform tasks with no per-task baselines to compare against. Each case reports a single successful run, with no repeated runs or ablations; the 6 to 33 minutes given per case count only the proposer session that found the answer, the paper does not name the language model behind the agents, memorized scientific knowledge cannot be ruled out on these public benchmarks, and the repository has the prompts and scheduler but not the nine runs. [code]
⚖️ Part III: Cross-Cutting Concerns
📊 Evaluation & Benchmarks (62)
Corresponds to §9 (Evaluation and Benchmarks).
Show 62 papers
GAIA: a benchmark for General AI Assistants (Mialon et al., ICLR 2024) - Reference benchmark for generalist, tool-using agent assistants; underlies popular public leaderboards tracking frontier agent progress.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (Jimenez et al., ICLR 2024) - De facto standard benchmark for coding/software-engineering agents; spawned the SWE-bench Verified/Lite/Live/Multimodal family. [code]
MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation (Huang et al., arXiv 2023) - Seminal benchmark for the 'AI research agent'/'ML engineering agent' evaluation subfield, precursor to MLE-bench, RE-Bench, etc. [code]
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., arXiv 2024) - Pioneered evaluation of agent-user interaction and policy compliance; standard reference for enterprise/customer-service agent evaluation. [code]
AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (Ma et al., NeurIPS 2024) - Widely-used unified evaluation toolkit addressing the coarse pass/fail scoring gap, directly relevant to evaluation-methodology framing. [code]
SmartPlay: A Benchmark for LLMs as Intelligent Agents (Wu et al., ICLR 2024) - Capability-decomposed evaluation methodology that influenced later fine-grained agent capability benchmarks. [code]
TravelPlanner: A Benchmark for Real-World Planning with Language Agents (Xie et al., ICML 2024) - Widely-cited stress test for complex constrained multi-tool planning, illustrating agents' distance from reliable long-horizon planning. [code]
InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback (Yang et al., NeurIPS 2023) - Established the interactive, execution-feedback evaluation paradigm underlying later coding-agent and terminal-agent benchmarks. [code]
GTA: A Benchmark for General Tool Agents (Wang et al., NeurIPS 2024) - Addresses realism gaps (implicit intent, authentic multimodal context) left by earlier synthetic tool-use benchmarks. [code]
Survey on Evaluation of LLM-based Agents (Yehudai et al., arXiv 2025) - Directly on-topic survey providing a ready-made taxonomy for a new agent survey's evaluation section.
Evaluation and Benchmarking of LLM Agents: A Survey (Mohammadi et al., KDD 2025) - Independently-developed sub-topic survey complementing Yehudai et al., useful for cross-checking taxonomy coverage.
τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., arXiv 2025) - Extends τ-bench to settings where user and agent both act on the environment. [code]
Agent-as-a-Judge: Evaluate Agents with Agents (Zhuge et al., arXiv 2024) - Systematizes agentic evaluation of agents; the reference point for judge-circularity concerns. [code]
Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (Kapoor et al., arXiv 2025) - Standardized harness that re-evaluates agents at scale with cost reported alongside accuracy.
Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reports (Yao et al., arXiv 2025) - Evaluates deep-research agents from answers to reports across semantic quality, topical focus, and retrieval trustworthiness. [code]
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (Merrill et al., arXiv 2026) - Hard, realistic command-line tasks; a de facto standard for terminal agents. [code]
Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents (DataAgentBench) (Ma et al., arXiv 2026) - Benchmark across heterogeneous database systems; frontier models reach only 38% accuracy on complex data questions. [code]
AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts (Li et al., arXiv 2026) - 32 real-world long-horizon scenarios testing autonomous agents under up to 1M-token contexts. [code]
Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation (Pysklo et al., arXiv 2026) - Benchmarks agents on enterprise API tasks via sandboxed code execution with state-diff (not trace-matching) success criteria. [code]
When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents (ToolMaze) (Zhu et al., arXiv 2026) - DAG-based benchmark with a perturbation taxonomy testing agent replanning and recovery when tool calls fail. [code]
Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values (Dong et al., arXiv 2026) - First dedicated agent-values benchmark: 394 executable environments and 4,335 value-conflict tasks across 28 value systems.
How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks (Huang et al., arXiv 2026) - Replay analysis asking how many tasks a benchmark actually needs before its ranking is trustworthy. [code]
Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents (Patel et al., arXiv 2026) - Argues agent leaderboards need predictive validity, not just a score, to say anything about deployment.
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments (Xi et al., arXiv 2026) - Benchmarks agents in de-idealized environments, closing the gap between clean benchmark worlds and real ones. [code]
Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents (Yi et al., arXiv 2026) - Measures how much the harness alone shifts an agent's beliefs across steps, isolating a confound in agent evaluation. [code]
Rethinking the Evaluation of Harness Evolution for Agents (Wang et al., arXiv 2026) - Rethinks how harness evolution should be evaluated, given that the harness moves the number as much as the model. [code]
ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions (Gupta et al., arXiv 2026) - Benchmarks agent reliability under production-like stress rather than single clean runs.
AgentAtlas: Beyond Outcome Leaderboards for LLM Agents (Mazaheri et al., arXiv 2026) - Moves agent evaluation beyond outcome leaderboards toward per-decision diagnosis of where control breaks.
CUBE: A Standard for Unifying Agent Benchmarks (Lacoste et al., arXiv 2026) - Proposes a standard for unifying heterogeneous agent benchmarks under one interface. [code]
UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (Chen et al., arXiv 2026) - Introduces UniClawBench, a capability-driven bilingual benchmark of 400 tasks executed in live Docker environments with step-wise checkpoints, evaluating proactive LLM agents across skill usage, exploration, long-context reasoning, multimodal understanding and cross-platform coordination, with a hidden supervisor agent so that multi-turn feedback never leaks the grading criteria. [code]
PolyWorkBench: Benchmarking LLM Agents for Cross-Lingual Long-Horizon Workflows (Li et al., arXiv 2026) - Introduces PolyWorkBench, a benchmark of 67 tasks spanning commerce, knowledge work, legal analysis, localization, and manufacturing that evaluates LLM agents on multilingual long-horizon workplace workflows via a hybrid of rubric-based structural scoring, executable state checks and an LLM judge; performance varies widely across languages and drops sharply on the harder cross-lingual tasks.
MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations (Shi et al., arXiv 2026) - Gives an agent an online store for 365 simulated days, built on 98,843 real product records from the 1688 wholesale marketplace, with 26 tools, refunds and bad reviews that surface days after the order, and final net assets as the score: across 48 runs of eight LLMs under a minimal ReAct loop and the Hermes agent framework, the best setup, Qwen3.7-Max with Hermes, ends with 27.3% of the mean net assets of three human participants with no e-commerce experience, a rule-based script that checks the store once a day beats 6 of the 16 setups, and Kimi K2.6 under Hermes decides on day 104 that the store cannot recover and then takes no store action in 355 of its remaining 523 decision windows; the released code runs on a synthetic catalog of 1,000 products because the real records and daily market reports are not redistributed, so the paper's numbers cannot be reproduced from the release alone. [code]
EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer (Gao et al., arXiv 2026) - Introduces a benchmark that evaluates LLM agent self-evolution as procedural-knowledge transfer, extracting trace-grounded "Abilities" from agent executions and organizing them into domain-specific Ability Graphs across web research, algorithmic reasoning, software engineering and knowledge work; curated Abilities transfer reliably across model families, but no automatic method gains in every setting.
Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle (Wang et al., arXiv 2026) - Introduces AARRI-Bench, a containerized benchmark suite evaluating whether frontier LLMs and agentic harnesses can perform entry-level research-intern tasks across the research lifecycle. [code]
ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences (Nguyen et al., arXiv 2026) - Introduces ReplicatorBench, a benchmark of human-verified replicable and non-replicable claims for evaluating LLM agents on replicating social and behavioral science research across data retrieval, experiment design/execution, and interpretation of results; agents design and run the experiments well but struggle to retrieve the new data a replication needs. [code]
Benchmark Test-Time Scaling of General LLM Agents (Li et al., arXiv 2026) - Introduces General AgentBench, a unified benchmark for evaluating general-purpose LLM agents across search, coding, reasoning, and tool-use, and finds that sequential and parallel test-time scaling fail to improve performance due to a context ceiling in the sequential case and a verification gap in the parallel one, while ten leading agents degrade substantially when moved from domain-specific evaluations into the general setting. [code]
BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks (Tu et al., arXiv 2026) - Turns frontier models on the benchmarks themselves, finding 12 author-confirmed issues in ScienceAgentBench including errors that made tasks unsolvable, and matching 83.3% of expert-identified issues on BIXBench at under USD 15 per 50-task audit.
Automated Benchmark Auditing for AI Agents and Large Language Models (Wang et al., arXiv 2026) - Introduces Auto Benchmark Audit (ABA), an agentic framework that audited 168 benchmarks across nine domains and found critical issues (ambiguous design, execution conflicts, incorrect ground truths) in over 25.7% of tasks; filtering those tasks out reshuffles model rankings and raises average scores on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6%.
PerspectiveGap: A Benchmark for Multi-Agent Orchestration Prompting (Sun et al., arXiv 2026) - Isolates orchestration prompt writing as its own capability: 110 scenarios over 10 topologies on which 33 models average a 17.2% pass rate. [code]
ClawBench: Can AI Agents Complete Everyday Online Tasks? (Zhang et al., arXiv 2026) - Puts browser agents on 153 everyday tasks across 144 live production sites, intercepting the final request so nothing is actually bought or booked; the strongest model tested clears a third of them. [code]
Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI (Shao et al., arXiv 2026) - Audits 2,385 traces across 15 agent benchmarks and finds exposures and reward hacking in about two thirds of Frontier Science and AutoLab tasks, inflating scores by 0.45 to 1.00.
The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation (Yu et al., arXiv 2026) - Agent configurations with identical accuracy differ 15.7x in bytes left on disk, so persistent storage belongs in the report alongside accuracy and reconstructability.
OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding (Zhou et al., arXiv 2026) - Prices 100 long-horizon office tasks by the human labor they replace, averaging 2.32 hours, and finds frontier models far cheaper than workers yet well short of their deliverable quality. [code]
AV-AIVAT: 74x Cheaper Agent Evaluation with Certified Anytime-Valid Stopping in Imperfect-Information Games (Li et al., arXiv 2026) - Lets an agent comparison stop as soon as the evidence settles without voiding the stated confidence level, pairing variance reduction with continuously monitored confidence sequences; over 71,439 paired poker hands, raw outcomes need a median 74x as many games.
PATH-Bench: Path-Dependent Evaluation of Lifelong Agents (Yang et al., arXiv 2026) - Evaluates lifelong agents on the order of their experience rather than task by task, measuring forward transfer, backward transfer and forgetting, and finds that strong transfer does not imply retention and that later experience can undo earlier gains.
Benchmarking LLM Judges for Mobile Agent Evaluation (Wang et al., arXiv 2026) - Tests six LLM-as-judge methods against 931 human-annotated mobile agent trajectories and finds a simple sampled-screenshot baseline competitive with or better than purpose-built judges, with the backbone model, not the pipeline, driving judge quality, and two backends showing opposite conservative and permissive failure profiles.
OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents (Caciolai et al., arXiv 2026) - Expanding GAIA2 into ten languages across five writing systems exposes a cross-lingual gap of 8.8 to 18.4 pass@3 points that concentrates on tool orchestration rather than quantitative reasoning and does not close with model scale, with error attribution assigning 55% of it to the model and bounding translation contamination at 6.4% of scenario-language pairs.
LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering (Wang et al., arXiv 2026) - Scores the model that steers rather than the model that codes: after each coding round the Controller under evaluation reads a structured run summary and tells a separate fixed Worker what to do next, what to verify, or whether to stop, across three settings trading execution scope against cost; the best strict success rate on full tasks is 24.69 per cent, and the cheap setting that never runs the Worker orders Controllers almost identically to the expensive one at a Spearman correlation of 0.97. [code]
τ^τ-Bench: An Environment for End-To-End, Realistic Agent Construction (Shi et al., arXiv 2026) - Makes building the agent the task: a developer agent inherits a codebase, a production API, a client holding the requirements and a cap on serving spend, then ships a customer-service agent that is scored by deploying it against held-out simulated users, and across 53 tasks the strongest configuration passes 23.9 per cent of evaluation simulations against an expert-authored ceiling of 82.2 per cent, failing in recognizably human ways by querying the records shallowly, telling the client almost nothing and shipping the first architecture that runs.
Autonomous Evaluation and Refinement of Digital Agents (Pan et al., COLM 2024) - Builds agent evaluators at several cost points that agree with oracle metrics 74.4 to 92.9% of the time, then uses them as rewards to improve the state of the art on WebArena by 29% without extra supervision. [code]
JEV-as-a-Judge: Accept When Confident, Escalate When Unsure (Li et al., arXiv 2026) - A judge that returns a decision rather than text comes within three points of the strongest LLM judge on ordinary preference and factuality at 0.36% of its fee, falls further behind when a derivation must be checked or a wrong answer is well written, and an accept-or-escalate cascade keeps 99% of the stronger judge's accuracy.
TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces (Min et al., arXiv 2026) - Builds 107 benchmarks from 252,557 Claude Code and OpenClaw deployment sessions by cutting each trace where the original agent showed an unwanted behavior and grading a model's next turn against a behavior-specific rubric, with no environment replay: nine frontier LLMs pass 26.7% on average, 67.9% when the rubric asks only for a well-formed tool call but 8.1% when it asks for a required check before continuing, and responses that open with a TodoWrite planning call pass 4.6% against 30.8% for other responses on the same instances, which the authors report as an association; the traces are not released, so the benchmarks cannot be rebuilt from public data.
TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution (Yao et al., arXiv 2026) - Audits coding-benchmark trials that pass the official tests: a GPT-5.5 auditor writes tests for each trial aimed at requirements in the prompt that the patch may break, and keeps only tests the reference patch passes, so every refutation comes with a test that can be replayed; across 6 model backends and 5 benchmarks it refutes 34.4% of the trials judged correct, lowering the resolution rate from 50.6% to 33.2%, with refutation rates from 7.6% on SWE-bench Verified to 53.6% on SWE-Marathon, though the auditor and its reviewer are LLMs, so a refutation is only as good as their reading of the prompt.
AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks (Lù et al., arXiv 2026) - Tests LLM judges of long computer-use tasks on 1,373 instruction and trajectory pairs from 166 hours of human recordings on three operating systems, where each negative pairs a recording with the instruction written for a closely related task, and runs eleven judges both on the full trajectory and as coding agents in five harnesses: GPT-5.5 in Codex is best at 80.9% balanced accuracy against 68.6% when prompted directly, while Qwen 3.6 27B scores 7.2 points lower inside OpenCode than without it and, given no instruction, calls 98.9% of trajectories successful; the negatives are near-miss swaps rather than real agent failures. [code]
Finding the Right Fit: Model-Harness Interactions across Agent Tasks (Li et al., arXiv 2026) - Pairs five models with four configurable harnesses, OpenHands, DeepSeek Harness, PI and openJiuwen, on TUA-Bench, ALE-CLI and Terminal-Bench 4, and adds the native Codex and Claude Code pairings, for 66 configurations: model rankings reverse across harnesses, with Claude 7.94 points ahead of GPT in OpenHands on Terminal-Bench 4 but 30.16 points behind in PI, the best harness changes between benchmarks for four of the five models, a model's own vendor harness is not reliably its best, and GPT-6 Astra scores 60.32% in PI at $4.66 per task against 52.38% in DeepSeek Harness at $19.94; each task counts once, from its last run. [code]
Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym (Kaesberg et al., arXiv 2026) - Turns 2D grid pathfinding with seven kinds of interacting rule cells into a Gymnasium task in which the agent draws the path one move at a time from the legal moves it is offered, optionally with backtracking, and tests eight open-weight models on 500 puzzles: the best, GPT-OSS 120B, solves 16.0% against 98.0% for people and 6.4% for a shortest-path agent that ignores the rules, the step-by-step format adds up to 5.4 points over one-shot answers for weaker models by removing formatting errors but costs stronger ones up to 5.6, backtracking raises GPT-OSS's completion rate from 85% to 94% while its solve rate falls 5.8 points, and Qwen3-VL-32B given the puzzle as an image solves 2.8% against 10.2% from text; no proprietary model is tested. [code]
Super Research: Answering Highly Complex Questions with Large Language Models through Super Deep and Super Wide Research (Dong et al., arXiv 2026) - Asks 300 open research questions in ten domains, drafted with GPT-4o and Gemini Deep Research and then vetted by domain experts, and scores each report against a reference graph of facts and insights that agents built and experts refined, on coverage, logical consistency, a quiz an LLM answers from the report alone, objectivity and citation spread: Gemini Deep Research leads 12 systems at 28.62 overall, Kimi-k2 answering with its built-in search (26.16) beats o3 Deep Research (22.04), and when 1 to 3 facts are removed from or added to a report the graph-based scores move the right way in 57.4% to 79.6% of cases against 14.8% to 22.2% for an LLM judge; the release has the questions and 3,600 reports but neither the reference graphs nor a scorer, so a new agent cannot yet be scored the way the paper does. [code]
SciΠvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents (Sun et al., arXiv 2026) - Builds 248 two-way choices from six months in which senior researchers ran LLM agents on 26 projects in six fields and corrected their research decisions: given the project record from just before a correction, a model picks between the agent's original plan and the corrected one. At medium reasoning effort four of five frontier models pick the original plan more often than not, Claude Opus 5 at 38.9% and GPT-6 Astra, the exception, at 59.5%; the best score in any setting is 63.5%, against 73.4% for 13 PhD students and postdocs new to the projects who had an agent assistant, and at that effort every model scores lower with the project record than with the two plans alone. The cases were kept because those same five models split on them, so the scores show where models go wrong rather than how often, and the questions, prompts and project records are released, without the models' response logs. [code]
Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution (Huang et al., arXiv 2026) - Builds 128 executable digital-workspace tasks from six agent benchmarks, marks 3,385 sensitive items in 100 of them, some already in the workspaces and some inserted by an LLM, and scores every run on completion, cloud cost and the share of those items the cloud model gets to see, comparing four ways of splitting the work between Qwen edge models and GLM-5.3 in the cloud. With Qwen3.5-2B, routing each step by the edge model's token entropy lifts completion from 12.75 to 39.97 at 22.7% of cloud-only cost, while letting the edge model call the cloud itself adds only 2 to 3 points; with Qwen3.5-9B, cloud advice that the edge model asks for reaches 60.46 against 82.70 for cloud-only at about 15% of its cost but shows the cloud 45.46% of the sensitive items; with Qwen3.7-Plus the four ways differ by 3.67 points in completion and by 66.23 in exposure. The tasks were filtered with runs of the same 2B and 9B edge models and GLM-5.3, dropping those both ends solved and those neither made headway on, nothing is redacted before a cloud call, and the release covers three of the four ways, without the delegation mode or the run logs. [code]
Measuring AI Agents as Teammates with Collaborative Intelligence (Wu et al., arXiv 2026) - Puts the model under test in a team chat of 2 to 5 with live LLM partners and one scripted teammate who creates the problem, over 240 tasks in eight dimensions of teamwork built from accident reports, clinical cases, negotiation datasets and coding benchmarks, and scores each model only on the tasks it solves alone. Every one of 18 models scores 20.2 to 46.6 points lower in the test than in a team where nothing goes wrong, and the best is 82.2, Gemini 4 Argon at high reasoning effort; frontier models pass at least 93% of the tests on refusing unsafe shortcuts and on reporting misconduct, but in a conflict they ask what the pushy teammate needs in 99% of runs and then accept its weaker split in 81%, and they pass 81% of tasks taken over from a weaker model that had cheated, against 97% when they start those tasks themselves. The scripted teammate speaks fixed lines and branches on how LLM judges label each reply, the live partners are LLMs rather than people, the alone and baseline versions are run once each, and 80 of the 240 tasks are held out and scored by the authors on request. [code]
TeamGym: A Gym for Team Decision-Making by AI Agents with Distributed Information (Park et al., arXiv 2026) - Builds team decision problems from 12 legal, finance, medical and software-engineering corpora, finds for each the smallest set of facts that together settle the reference decision while each one is needed, and controls which participant starts with which fact. Averaged over eight models, splitting those facts across participants lowers success from 63.4% for one model holding them all to 40.5%, and making participants find them in their own source documents lowers it to 20.6%; GRPO training of Qwen2.5-7B, with a GPT-5.5 judge scoring only the final decision, raises FinDVer validation from 21.85% to 41.08% and the share of episodes in which every participant speaks from 61.7% to 86.7%, which the reward does not count. The facts were picked and checked with models, several decisions are graded by an LLM judge, each training result comes from one run, gains on CUAD barely carried over to MAUD, and the code is not yet released.
TeamBench: Evaluating Agent Coordination under Enforced Role Separation (Kim et al., arXiv 2026) - Runs Planner, Executor and Verifier in separate containers so that no role can read the full requirements, edit the workspace and certify the result all at once, over 851 task templates, 650 of them adapted from GitHub bug reports, each with a deterministic grader. On its 90-task leaderboard the best of 13 models, Claude Opus 4.7, passes 37.8% as a team against 35.6% alone; with Gemini-3 Flash in every role on 155 tasks, the team beats a single agent by 15.7 points on the fifth of tasks the single agent finds hardest but trails it by 6.8 to 10.1 on the three fifths it finds easiest, and dropping the Verifier raises the mean partial score by 5.5. Across the role-mixing runs, Verifiers that return a verdict approve 49.4% of the submissions the grader rejects, 22.3% if runs without a verdict are counted too, and prompt-only roles reach a pass rate statistically indistinguishable from enforced ones while the Verifier tries to edit the Executor's code 3.6 times as often. Each role works in one file-based pass with capped turns and no multi-round dialogue between roles, and open-weight models of 30B or less fail mostly on tool calls, so for them the benchmark measures tool-call reliability more than coordination. [code]
🛡️ Safety & Alignment (62)
Corresponds to §11 (Safety, Security, and Trustworthiness).
Show 62 papers
Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (Greshake et al., arXiv 2023) - The founding paper of the indirect prompt injection threat model that underlies almost all later LLM-agent security research. ⭐ [code]
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents (Debenedetti et al., NeurIPS 2024) - The most widely used standardized testbed for measuring agent robustness to prompt-injection attacks and defenses. [code]
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents (Zhan et al., ACL 2024) - Standard reference benchmark for quantifying tool-integrated agents' susceptibility to indirect prompt injection. [code]
WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks (Evtimov et al., arXiv 2025) - Extends prompt-injection evaluation from single-step tool calls to realistic, multi-step autonomous web-browsing agents. [code]
R-Judge: Benchmarking Safety Risk Awareness for LLM Agents (Yuan et al., EMNLP 2024) - Widely cited benchmark for evaluating LLMs' own risk-awareness/judgment capability as a safety-monitoring component for agents. [code]
Identifying the Risks of LM Agents with an LM-Emulated Sandbox (Ruan et al., ICLR 2024) - Foundational scalable methodology for red-teaming/risk discovery in tool-using agents without needing real-world tool access. [code]
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (Andriushchenko et al., ICLR 2025) - Key benchmark distinguishing agent misuse risk from chatbot jailbreak risk, showing agentic capability compounds harm potential. [code]
Evil Geniuses: Delving into the Safety of LLM-based Agents (Tian et al., arXiv 2023) - One of the earliest systematic studies showing multi-agent LLM collaboration amplifies rather than mitigates safety risk. [code]
BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents (Wang et al., ACL 2024) - Seminal demonstration that agent backdoors survive downstream safety fine-tuning, motivating agent supply-chain security concerns. [code]
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases (Chen et al., NeurIPS 2024) - Establishes memory/knowledge-base poisoning as a distinct, training-free attack surface unique to memory-augmented LLM agents. [code]
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (Zhang et al., arXiv 2024) - Largest unified taxonomy/benchmark integrating diverse attack and defense classes for LLM agents into one evaluation framework. [code]
TrustAgent: Towards Safe and Trustworthy LLM-based Agents (Hua et al., EMNLP 2024) - Influential early defense/mitigation framework proposing constitution-guided planning as a mechanism for agent safety. [code]
SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents (Yin et al., arXiv 2024) - Extends agent-safety evaluation beyond digital/text domains to physical-world embodied hazards, showing safety failures generalize to robotics. [code]
AI Agents That Matter (Kapoor et al., arXiv 2024) - Widely cited critique reshaping how the field evaluates agent capability claims, directly relevant to trustworthy assessment of agent risk/benefit tradeoffs.
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (Hubinger et al., arXiv 2024) - Landmark demonstration that current safety-training pipelines can fail to remove hidden deceptive/misaligned behavior, directly informing agent trustworthiness concerns. [code]
Frontier Models are Capable of In-context Scheming (Meinke et al., arXiv 2024) - First systematic empirical evidence of in-context scheming capability in frontier autonomous agents, a central concern for autonomy-related alignment risk.
Emergent Misalignment: Narrow Finetuning can Produce Broadly Misaligned LLMs (Betley et al., arXiv 2025) - Recent, highly influential finding that narrow, seemingly innocuous fine-tuning of agentic capabilities can produce broad, unpredictable safety failures. [code]
AI Deception: A Survey of Examples, Risks, and Potential Solutions (Park et al., arXiv 2023) - Foundational survey establishing AI deception as a distinct, empirically-grounded risk category central to agent trustworthiness and alignment.
AI Alignment: A Comprehensive Survey (Ji et al., arXiv 2023) - One of the most comprehensive general alignment surveys, providing the conceptual scaffolding (RICE, forward/backward alignment) that agent-specific safety work builds on. [code]
A Survey on Trustworthy LLM Agents: Threats and Countermeasures (Yu et al., arXiv 2025) - The most directly on-topic recent survey for this sub-area, providing a taxonomy spanning nearly all agent-specific safety/security threat classes. [code]
A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment (Wang et al., arXiv 2025) - Large-scale lifecycle-wide safety survey situating agent-specific risks within the broader LLM safety pipeline, useful for framing agent risk as one stage of a larger safety stack.
A Survey on Autonomy-Induced Security Risks in Large Model-Based Agents (Su et al., arXiv 2025) - Directly frames the core thesis of this sub-topic: that autonomy itself, not just the underlying LLM, is the source of new agent-specific security risks.
Discovering Language Model Behaviors with Model-Written Evaluations (Perez et al., arXiv 2022) - Early, seminal empirical evidence connecting RLHF training scale to emergent self-preservation and power-seeking-adjacent expressed preferences, a precursor concern for autonomous agent alignment. [code]
Towards Understanding Sycophancy in Language Models (Sharma et al., arXiv 2023) - Key empirical study of sycophancy as a robustness/alignment failure mode with direct implications for agents that must give honest assessments during autonomous decision-making. [code]
Design Patterns for Securing LLM Agents against Prompt Injections (Beurer-Kellner et al., arXiv 2025) - Catalogue of architectural patterns that constrain what an agent may do after reading untrusted input.
OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents (Kuntz et al., arXiv 2025) - Extends agent-safety measurement to computer-use agents operating real interfaces.
The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems (Staufer et al., FAccT 2026) - Empirical index of 30 deployed agentic systems; documents a capability-vs-safety transparency gap.
AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security (Liu et al., arXiv 2026) - Unified taxonomy of agentic risks powering a diagnostic guardrail that flags root causes of unsafe trajectories instead of binary labels. [code]
The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis (Wang et al., arXiv 2026) - Systematizes indirect prompt-injection threats against LLM agents into a taxonomy and analyzes defense coverage gaps.
ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection (Weng et al., arXiv 2026) - Builds an influence-provenance graph auditing whether agent decisions rest on trustworthy evidence, cutting attack success to 3.8%.
Provably Secure Agent Guardrail (Wu et al., arXiv 2026) - Forces agents to formalize intentions as first-order-logic constraints before acting, reaching zero attack success and zero false positives.
AutoRISE: Agent-Driven Strategy Evolution for Red-Teaming Large Language Models (Gautam et al., arXiv 2026) - A coding agent evolves executable attack strategies (not just prompts), improving jailbreak attack success by 17 points across 11 models.
When Agents Remember Too Much: Memory Poisoning Attacks on Large Language Model Agents (Torres et al., arXiv 2026) - Memory poisoning attacks on agents: corrupting durable state so the compromise survives the session.
Agent Data Injection Attacks are Realistic Threats to AI Agents (Choi et al., arXiv 2026) - Shows data injection into an agent's inputs is a realistic threat, not a contrived laboratory setup.
AgentAbstain: Do LLM Agents Know When Not to Act? (Liu et al., arXiv 2026) - Asks whether agents know when not to act, treating abstention as a first-class safety behaviour. [code]
Prismata: Confining Cross-Site Prompt Injection in Web Agents (Villa et al., arXiv 2026) - Confines cross-site prompt injection in web agents at the boundary rather than trusting the model to resist it.
The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities (Rashidi et al., arXiv 2026) - Surveys the fragmented execution-security work for coding agents across isolation, access control, and related defences.
Supply-Chain Poisoning Attacks Against LLM Coding Agent Skill Ecosystems (Qu et al., arXiv 2026) - Demonstrates supply-chain poisoning attacks against the skill ecosystems that coding agents install from.
Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systems (Chang et al., arXiv 2026) - Studies indirect prompt injection in the wild, showing the retrieval barrier is weaker than assumed for real LLM systems.
PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses (Yin et al., arXiv 2026) - Reinforcement-learning red teaming that automatically stress-tests prompt-injection defenses. [code]
MOSAIC: Knowledge-Guided CLI Command Composition Attack in LLM Coding Agents (Wu et al., arXiv 2026) - Identifies a composition-level attack surface in LLM coding agents whereby individually benign CLI commands form hazardous producer-consumer state relations, and introduces MOSAIC.
KidnapRAG: A Black-Box Attack for Hijacking Reasoning in Agentic Retrieval-Augmented Generation Systems (Choi et al., arXiv 2026) - Black-box poisoning of agentic RAG through three role-specific documents, Bait, Chain-Link and Mal-Ins, which attract the first retrieval, induce query reformulation, and supply attacker-controlled evidence, with no access to prompts, traces or parameters. [code]
(A)I Sees What You Don't: Exploiting New Attack Surfaces in Third-Party Mobile Agents (Zhang et al., arXiv 2026) - Identifies two previously uncharacterized attack surfaces in VLM-driven third-party mobile agents, the screen perception surface arising from human-machine vision discrepancies and the misused channel surface that intercepts or manipulates the agent's execution pipeline; across five popular frameworks, seven concrete attacks let a malicious app with no special permissions hijack agent actions, up to arbitrary command execution.
Beware of Agentic Botnets: Scalable Untargeted Promptware Attacks via Universal and Transferable Adversarial HalluSquatting (Spira et al., arXiv 2026) - Introduces adversarial hallucination squatting, in which attackers model the distribution of LLM-hallucinated resource identifiers for trending repositories and skills and preemptively register those names to host adversarial prompts; hallucinated identifiers appear in up to 85% of repository-cloning and up to 100% of skill-installation scenarios, transfer across models, and yield remote code execution in production LLM applications.
BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agents (Feng et al., arXiv 2026) - Presents a stage-aware framework and benchmark that instruments LLM-agent workflows (planning, memory, tool-use) to inject, track, and evaluate backdoor triggers across multi-step trajectories. [code]
Defense Against Indirect Prompt Injection via Tool Result Parsing (Yu et al., arXiv 2026) - Proposes a tool-result-parsing defense that extracts and sanitizes tool outputs to reduce the attack success rate of indirect prompt injection on LLM agents while preserving task utility, evaluated on the AgentDojo benchmark. [code]
ICON: Indirect Prompt Injection Defense for Agents based on Inference-Time Correction (Wang et al., arXiv 2026) - Proposes an inference-time defense for LLM agents that detects indirect prompt injection via latent-space analysis and neutralizes it through attention manipulation while preserving task utility.
An AI Agent Execution Environment to Safeguard User Data (Stanley et al., arXiv 2026) - Introduces GAAP, an AI agent execution environment that uses information flow control to enforce user-specified permissions on how private user data is disclosed, including to the AI model and its provider.
Protocol-Level Attacks on Agentic Commerce Platforms: A Cross-Platform Taxonomy, AIP-Bench, and Unified Defense (Louck, arXiv 2026) - Moves agentic-commerce security down a layer: 33 protocol vulnerabilities across three platforms exploit deterministically regardless of the model, and three of them chain into an end-to-end payment hijack.
IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests (Singh et al., arXiv 2026) - 66.5% of malicious issue requests slip past every guardrail in deployed Cursor, Claude Code and Codex Desktop, and the refusals that do happen come from the model, not the agent framework.
Rethinking MCP Security: A Large-Scale Study of Runtime MCP Servers and Security Scanner Reliability (Chen et al., arXiv 2026) - Collects 64,611 in-the-wild MCP servers, over 37,000 of them runnable, and finds the scanners auditing them cry wolf: fewer than half of sampled alerts survive manual validation.
Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming (Wang et al., arXiv 2026) - Builds a transferable prompt-injection strategy library instead of an RL attacker fitted to one target, then reuses it untrained on unseen models: 76.2% attack success against Gemini-2.5-Pro and 86.7% on AgentDojo from roughly ten queries per sample. [code]
LoginTrap: Uncovering Task-Agnostic Phishing-Style Indirect Prompt Injection Attacks against LLM-based Web Agents (Guo et al., arXiv 2026) - Shows a web agent can be talked into logging in: page content the attacker controls makes authentication look like a prerequisite for the task and steers the agent to a controlled login page, reaching 86% average end-to-end success without knowing the user's task.
Emergent Misaligned Communication in Long-Horizon Multi-Agent LLM Commerce (Li et al., arXiv 2026) - Across 2,583 inter-agent emails from 20 year-long vending simulations spanning 13 frontier models, 12.6% contain false factual claims, manipulation, collusion or threats without any engineered elicitation, the behaviour is reciprocal (1.65x odds of a misaligned reply) and stress-conditioned (1.58x under low inventory), and model performance rank does not predict it.
Governance at the Boundary: How Agent Decomposition Degrades Policy Compliance (Li et al., arXiv 2026) - Decomposing an agent degrades its governability at the handoff boundary: over 626 KYC/AML episodes a 32B open-weights model attenuated 0% of policy-relevant discovered facts under a single loop, 56% under a fixed pipeline and 85% under an orchestrator-subagent architecture, and the same mechanism produces both under-escalation and over-escalation.
What's in Your Agent's Context? Context Privilege Escalation Attacks against AI Agent Harness (Li et al., arXiv 2026) - Systematizes how real harnesses assemble context and names two escalation paths that fall out of those designs: attacker-controlled content from a low-privileged source landing in a higher-privileged message role, and attacker-controlled content persisting past the scope it entered in, demonstrated against 12 harnesses including Claude Code and Codex with consequences running to remote code execution and manipulated tool or skill invocation.
BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks (Prasad et al., arXiv 2026) - Plants an optional shortcut in each of three synthetic tabular ML tasks that inflates the public score while failing a hidden test set and breaks no stated rule, and 57.1 per cent of runs across seven frontier agents take it, five of the seven above half, with the mean staying above half even when the prompt asks them not to cheat.
LlamaFirewall: An open source guardrail system for building secure AI agents (Chennabasappa et al., arXiv 2025) - Stacks a small jailbreak classifier, a chain-of-thought auditor for injection and goal drift, and a static code scanner as the last runtime layer around an agent; on AgentDojo the 86M classifier alone cuts attack success from 17.6% to 7.5%, and adding the auditor brings it to 1.75%. [code]
Type-Safe Is Not Error-Free: A Constrained Decision Head Follows the Option Name, Not the Rubric Bound to It (Sun and Xu, arXiv 2026) - Swaps which option name is bound to which rubric and finds typed decision models follow the name: renaming 0/1 to no/yes flips 70.4 more answers per hundred on 1,200 workflow decisions, Jev itself shows the same reversal, and the type-error rate stays at 0% throughout.
Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents (Remedios et al., arXiv 2026) - Red-teams the blocking monitors that approve each action of production coding agents, Auto Mode in Claude Code and an approximation of Guardian in Codex, with an Opus 5 attacker told how the monitor works: injections the agent writes into its own actions get past Auto Mode in 78.4% of rollouts and Guardian in 85.2%, mean attack success over four attack types is 65% and 72%, and Auto Mode ++, with escaped and framed transcripts, monitoring of every edit and of compaction, and an agentic final stage, lowers it to 9% while blocking under 1% of 4,000 benign actions, though attacks spread across several contexts remain costly to stop. [code]
ClashBench: Conflicts Leading Agents to Seize and Harm (Xie et al., arXiv 2026) - Gives agents a task whose port, lock, disk, GPU memory or room booking is already held by a running task, in 268 executable cases over 55 resource types, and runs 17 models through Codex, Claude Code and OpenCode: in 44.5% of runs the agent completes its task while the incumbent fails its health check, from 25.2% for Claude-Sonnet-5 to 61.2% for Qwen3.8-Max, both in Claude Code, agents often take over after recognizing the conflict, the final reply mentions neither the conflict nor the takeover in 31.9% of these cases, and an instruction to leave existing work alone still leaves 38.2%; most cases were run once. [code]
HarnessSecurity-Bench: Do Security Mechanisms Really Protect Coding Agent Harnesses? (Zhu et al., arXiv 2026) - Rates ten security mechanisms across 40 coding-agent harnesses and then tests nine of them in 2,500 trials of 23 tasks seeded with misleading context and malicious resources, on Claude Code, Codex CLI, Gemini CLI, gptme, Qwen Code and GitHub Copilot, all driven by GLM-5.2: auto-approve raises utility from 77.1% to 95.4% and attack success from 29.2% to 95.6%, network isolation and read-only mode cut attack success to 0.8% at a cost of 24.5 and 34.0 utility points, and command allowlisting cuts it by 48.1 points for a 1.5-point loss, though an allowed Python interpreter once got around it; one backbone only. [code]
🔗 Related Awesome Lists
Other useful reading lists in the same space:
- js-lee-AI/awesome-agent-loop-papers: the list for the follow-up survey, on the agent loop itself: control strategies, trained loops, skills, harnesses, and the evaluation and safety problems that come with the loop.
- Hannibal046/Awesome-LLM: the canonical hub of large language model papers, tools, and resources.
- ysymyth/awesome-language-agents: language-agent reading list organized around the CoALA framework.
- WooooDyy/LLM-Agent-Paper-List: survey-backed agent paper list from The Rise and Potential of LLM-Based Agents (Fudan NLP).
- luo-junyu/Awesome-Agent-Papers: taxonomy-driven survey companion on agent construction, collaboration, and evolution.
- e2b-dev/awesome-ai-agents: a large, media-rich directory of open- and closed-source agent products & frameworks: tools for building agents rather than papers.
- kyrolabs/awesome-agents: curated agent frameworks and libraries, each with a live star badge.
Maintain a related list? Open a PR to add it here; we're happy to cross-link.
📄 Citation
The survey is posted on Preprints.org as LLM Agents: A Survey, DOI 10.20944/preprints202608.0265.v1. That versioned record is the one to cite. The same 47-page paper is also in this repository at paper/llm-agents-a-survey.pdf so you can read it without leaving GitHub.
If this list or the survey is useful to you, please cite:
@article{lee2026llmagents,
title = {LLM Agents: A Survey},
author = {Lee, Jungseob},
year = {2026},
month = {August},
journal = {Preprints},
publisher = {Preprints},
doi = {10.20944/preprints202608.0265.v1},
url = {https://doi.org/10.20944/preprints202608.0265.v1}
}
GitHub's Cite this repository button reads CITATION.cff and returns the same record in APA or BibTeX.
The follow-up survey, The Agent Loop: A Survey of Control Strategies, Skills, and Harnesses for LLM Agents, has its own record: DOI 10.2139/ssrn.7186738. Cite whichever one you actually drew on.
🤝 Contributing
The agent literature is arriving faster than any one person can track, on the order of a thousand new papers a month. I read a lot for this list, but I have certainly missed strong papers and methods. If one belongs here (including your own), please help:
- Open a PR adding it to the right section with a verifiable link and a one-line why it matters (and a
[code]link if an implementation exists), or - Open an issue with the link and I'll triage quickly.
Corrections, sharper annotations, and whole new sections are just as welcome. See CONTRIBUTING.md for the entry format.
Most of the languages this list is translated into aren't my first language. If a sentence in one of the translations reads oddly or a term is translated wrong, please open an issue, or fix it directly with a PR against that file.
👥 Contributors
This list is community-maintained. Thanks to everyone who has suggested, verified, or annotated a paper:
Show 24 contributors
| Contributor | Contributed | |
|---|---|---|
| @ybkim95 | The Collaborative Intelligence Benchmark (a model as a teammate, scored only on tasks it solves alone), TeamGym (team decisions with the deciding facts split across participants) and TeamBench (Planner, Executor and Verifier kept apart by container permissions), in Evaluation & Benchmarks (#34). Submitted by one of their authors, the first author of TeamBench | |
| @passionate11 | EC2-Bench, 128 workspace tasks that score edge-cloud collaboration on completion, cloud cost and privacy exposure, submitted by its first author, in Evaluation & Benchmarks (#33); ToolMaster, which trains a model to try tools out before relying on them, submitted by one of its two first authors, in Tool Use (#28) | |
| @ChaoYue0307 | Harness Engineering for Language Agents, a position paper that splits the harness into control, agency and runtime and proposes HarnessCard for reporting it, submitted by its first author, in Surveys & Position Papers (#27) | |
| @WhymustIhaveaname | AgonAlpha, alpha mining whose reviewer reruns each candidate and can veto it, and AutoSR, symbolic regression run as a search over research states, both submitted by one of their co-first authors, in Applications (#32); SciΠvot, a benchmark of research decisions built from senior researchers' corrections of LLM agents, submitted by its first author, in Evaluation & Benchmarks (#29); 3 research-agent and orchestration papers: AutoNumerics, OptimAI, PerspectiveGap (#1); the Agon code link, and the report that led to nine truncated glosses being rewritten (#5) | |
| @dolev31 | TabAgent, which replaces an agent's repeated tool-shortlisting call with a small classifier trained on its own runs, submitted by its first author, in Tool Use (#30) | |
| @segevshlomovIBM | From Benchmarks to Business Impact, IBM's pilot of its CUGA agent in talent acquisition, submitted by its first author, in Applications (#25) | |
| @KhanCold | MerchantBench, a benchmark that has an agent run an online store for 365 simulated days, submitted by its first author, in Evaluation & Benchmarks (#24) | |
| @cnsdqd-dyb | Super Research, a benchmark of 300 open research questions with reports from 12 systems, submitted by its first author, in Evaluation & Benchmarks (#23) | |
| @lkaesberg | Voting or Consensus? (decision protocols in debate), Stay Focused (problem drift in debate) and MALLM, the debate framework both ran on, in Multi-Agent Systems; Spatial-Gym, step-by-step spatial reasoning for agents, in Evaluation & Benchmarks (#22). Submitted by one of their authors | |
| @markovchain-builder | SELF, agent training from environment feedback without rewards, in Planning & Reasoning; EFRE, a repertoire of prompts for continual learning with a frozen LLM, in Memory (#21) | |
| @ZhishanQ | TraceDance, benchmarks built from real agent deployment traces, submitted by its first author, in Evaluation & Benchmarks (#20) | |
| @Gavin-M34 | Jev in the Wild, a study of how public GitHub projects use Jev, in Surveys & Position Papers (#19) | |
| @shubhamrgandhi | Steer, Don't Solve, a small critic model that steers a larger coding agent, in Planning & Reasoning (#16); DRACO, rubric-based credit assignment for agents trained without a verifier, in Tool Use (#17). Submitted by their first author | |
| @sunyuhan19981208 | The TaoLive report on training a compact model to keep working as its harness changes, submitted by one of its authors, in Applications (#15) | |
| @burgerseater | LoopArena, a benchmark that scores the model steering a coding agent rather than the one writing the code, suggested by one of its authors, in Evaluation & Benchmarks (#14) | |
| @dukesun99 | Corpus2Skill in Memory, OrchMAS in Multi-Agent Systems, and an IR-for-agents position paper in Surveys & Position Papers, submitted by an author on all three (#13) | |
| @zhongzero | ForeDreamer, a dual-agent memory architecture for forecasting, submitted by one of its authors, in Memory (#12) | |
| @Nicolas99-9 | CityReal, a large-scale human-aligned urban simulation, in Multi-Agent Systems (#10) | |
| @BobbyZhouZijian | CORAL, an autonomous multi-agent evolution framework, submitted by one of its authors, in Agent Architectures & Frameworks (#9) | |
| @razzant | Ouroboros, a self-developing coding-agent harness, submitted by its maintainer, in Agent Architectures & Frameworks (#8) | |
| @reacher-z | Dr. Bench, a deep-research-agent evaluation, submitted by one of its authors, in Evaluation & Benchmarks (#7); ClawBench, a live-web browser-agent benchmark, same section (#3) | |
| @JEONGSEJIN | WebAgent and the world-model-augmented web agent, in Interactive Environments (#6) | |
| @jinmang2 | 6 agent-memory systems: MemoryOS, Zep, Nemori, MemOS, G-Memory, ACE (#2) | |
| @js-lee-AI | Maintainer |
Listed by hand rather than generated, so that it credits the people who contributed, not whoever happened to run git commit. See the contributor graph for commit-level history.
Want your avatar here? See CONTRIBUTING.md: a single well-formatted PR is all it takes.
📜 License
Released under the MIT License.
🗓️ Updates
- 2026-11-06: The Collaborative Intelligence Benchmark, TeamGym and TeamBench added to Evaluation & Benchmarks by one of their authors (@ybkim95, #34). In TeamGym, splitting the deciding facts across a team instead of giving them all to one model lowers success from 63.4% to 40.5%. 580 to 583.
- 2026-11-05: EC2-Bench added to Evaluation & Benchmarks by its first author (@passionate11, #33). With Qwen3.7-Plus on the edge, the four ways of bringing in the cloud model differ by 3.67 points in completion and by 66.23 in how much of the sensitive data the cloud sees. 579 to 580.
- 2026-11-02: Harness Engineering for Language Agents added to Surveys & Position Papers by its first author (@ChaoYue0307, #27). In its audit of 63 works, the 38 papers and benchmarks most often foreground interfaces and observability, and the 25 engineering notes foreground runtime, control and governance. 578 to 579.
- 2026-11-02: AgonAlpha and AutoSR added to Applications by one of their co-first authors (@WhymustIhaveaname, #32). AutoSR finds the exact equation on three cp3-bench problems that none of 12 published symbolic-regression systems solved, from one successful run per problem. 576 to 578.
- 2026-10-28: TabAgent added to Tool Use by its first author (@dolev31, #30). A classifier trained on the agent's own runs takes over GPT-5.4's tool shortlisting on AppWorld at about 3 ms a call instead of 7.5 s, and a live agent's task success moves from 29.2% to 27.1%, a gap too small to tell apart on 24 tasks. 575 to 576.
- 2026-10-28: SciΠvot added to Evaluation & Benchmarks by its first author (@WhymustIhaveaname, #29). Asked to choose between an agent's research plan and the one a senior researcher corrected it to, the best frontier model reaches 63.5%, against 73.4% for researchers new to the projects. 574 to 575.
- 2026-10-28: ToolMaster added to Tool Use by one of its two first authors (@passionate11, #28). Trained to try tools before relying on them, Qwen2.5-7B reaches 66.4 on StableToolBench, against 60.2 for GPT-4o without training. 573 to 574.
- 2026-10-19: From Benchmarks to Business Impact added to Applications by its first author (@segevshlomovIBM, #25). In a talent-acquisition pilot that is not yet in full production, IBM's CUGA agent reaches 87% on the 26 tasks of the new BPO-TA set. 572 to 573.
- 2026-10-17: MerchantBench added to Evaluation & Benchmarks by its first author (@KhanCold, #24). After 365 simulated days of running an online store, the best of 16 LLM setups ends with 27.3% of the human participants' mean net assets. 571 to 572.
- 2026-10-16: Super Research added to Evaluation & Benchmarks by its first author (@cnsdqd-dyb, #23). On its 300 research questions the best of 12 systems, Gemini Deep Research, scores 28.62 out of 100. 570 to 571.
- 2026-10-15: Voting or Consensus?, Stay Focused and MALLM added to Multi-Agent Systems and Spatial-Gym to Evaluation & Benchmarks by one of their authors (@lkaesberg, #22). In Spatial-Gym the best of eight open-weight models solves 16.0% of the grid puzzles, against 98.0% for people. 566 to 570.
- 2026-10-14: +30 papers posted since mid-September 2026, three per section. Several report negative results; in one, 34.4 per cent of the coding-agent runs that benchmarks judge correct break a requirement stated in the task. 536 to 566.
- 2026-10-12: SELF added to Planning & Reasoning and EFRE to Memory (@markovchain-builder, #21). SELF trains agents without rewards, and with Qwen3-8B it reaches 43.2% on τ-bench Retail against 39.1% for GRPO trained on rewards. 534 to 536.
- 2026-10-03: TraceDance added to Evaluation & Benchmarks by its first author (@ZhishanQ, #20). It builds tests from real agent sessions, and on them nine frontier LLMs pass 8.1% of the cases that call for a check before continuing. 533 to 534.
- 2026-09-30: Jev in the Wild, a study of how 2,170 public GitHub projects use Jev, added to Surveys & Position Papers (@Gavin-M34, #19). 532 to 533.
- 2026-09-24: The list and the online book on GitBook are now also available in Brazilian Portuguese and Spanish.
- 2026-09-24: The survey is now on GitBook as an online book, written for readers new to the area.
- 2026-09-24: The list is now available in Korean, Simplified Chinese and Japanese; choose a language at the top. The notes and prose are translated, and paper titles and terms of art are left in English. While translating I found notes that stopped mid-sentence and entries missing their first author, and fixed them.
- 2026-09-23: Twelve papers on models inside the agent loop that answer with a typed decision instead of text and hand off to an LLM when they aren't sure. Jev's release drew wide attention to the idea, and four of the papers study Jev itself; in one of them, Jev follows an option's name rather than the rubric attached to it and never raises a type error. 520 to 532.
- 2026-09-16: Three community additions merged, all from authors of the papers: the TaoLive harness-aware training report (@sunyuhan19981208, #15), Steer, Don't Solve and DRACO (@shubhamrgandhi, #16 and #17). The glosses now include the papers' own numbers; the TaoLive one leads with the finding that fixed-harness finetuning costs 7.7 points of instruction following while harness-state augmentation costs none. 517 to 520.
- 2026-09-16: The contributors table is now collapsible like the sections, and
sync_counts.pyupdates its count as well. - 2026-09-07: +20 papers posted since May 2026, two per section. LoopArena was suggested by one of its authors, @burgerseater, in #14. Several are negative results; in one, frontier agents take a planted shortcut in 57 per cent of runs even when the prompt asks them not to. 497 to 517.
- 2026-09-03: Three papers added by @dukesun99, an author on all three (#13): Corpus2Skill to Memory, OrchMAS to Multi-Agent Systems, and Information Retrieval Misses the Mark for LLM Agents to Surveys & Position Papers. That last one is the list's first SSRN paper. The Corpus2Skill gloss now gives its results on eleven datasets, where corpus navigation loses on three. 494 to 497.
- 2026-08-25: ForeDreamer added to Memory by @zhongzero, who is one of its authors (#12). The gloss now starts with what sets the system apart: it turns web evidence into structured memory before forecasting instead of passing retrieval results straight to the agent. 493 to 494.
- 2026-08-20: +20 papers from August 2026, two per section. Several are negative results, such as a multilingual gap that doesn't close as models get larger. 473 to 493.
- 2026-08-20: Two community additions merged: Dr. Bench (@reacher-z, #7) and CityReal (@Nicolas99-9, #10). 471 to 473.
- 2026-08-12: Four community additions merged: Ouroboros (@razzant, #8), CORAL (@BobbyZhouZijian, #9), and WebAgent plus the world-model-augmented web agent (@JEONGSEJIN, #6). One code link was dropped that had been copied from the entry above it. 467 to 471.
- 2026-08-08: +16 papers from August 2026 across all ten sections, with official repos linked where they exist. 451 to 467.
- 2026-08-08: Nine glosses had been cut off mid-sentence by an earlier backfill. They're fixed, and Agon has the
[code]link that @WhymustIhaveaname supplied in #5. There's a check for this inscripts/check_glosses.pynow. - 2026-08-06: The survey is posted on Preprints.org with DOI
10.20944/preprints202608.0265.v1. The citation block,CITATION.cffand the header link now point at the versioned record instead of the PDF in this repository. - 2026-07-31: +30 papers from July 2026, three per section, with official repos linked where they exist. 421 to 451.
- 2026-07-26: ClawBench added to Evaluation & Benchmarks by @reacher-z, who helps maintain it. 420 to 421.
- 2026-07-25: First community contributions: +6 agent-memory systems (MemoryOS, Zep, Nemori, MemOS, G-Memory, ACE) from @jinmang2, and +3 research-agent and orchestration papers (AutoNumerics, OptimAI, PerspectiveGap) from @WhymustIhaveaname.
- 2026-07-19: +78 papers from January to July 2026 across all ten sections, with official repos linked where they exist.
- 2026-07-19: +30 papers from January to May 2026 (three per section), with official repos linked where they exist.
- 2026-07-16: +50 papers from June and July 2026 across all ten sections, with official repos linked where they exist.
- 2026-07-12: +42 papers from 2026 across all ten sections, a new 10 to Watch (2026) section with live star counts, and live star badges on the related lists. 211 to 253.
- 2026-07-09: +27 papers on agentic RL, protocols, deep research, and frontier evaluation and safety. 184 to 211.
- 2026-07-08: Initial release. 184 annotated papers, organized by the survey's taxonomy.