Skip to content
Projects

js-lee-AI/awesome-llm-agent-papers

View on GitHub

js-lee-AI/awesome-llm-agent-papers has 114 GitHub stars and 41 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-15 15:55 UTC. Last push 2026-11-14.

How these counts are measured

What you can do

  • 📚 A superset of the survey - The 228 references behind “LLM Agents: A Survey,” plus papers that came out after the paper was written. New papers are still being added.
  • 🧭 Organized by function - 10 sections following the survey's structure: Surveys, Architectures, Planning, Memory, Tool Use, Multi-Agent, Environments, Applications, Evaluation, Safety.
  • ✍️ Annotated - Each entry has a one-line note on what it contributes, its venue and year, and a [code] link where an official implementation exists.
  • ⭐ Starter Kit - A 10-paper list for getting oriented, with a note on why each one is worth reading first.

Measured star growth

in 24h
0
in 7 days
Update delayed
in 15 days
Update delayed
in 1 month
Update delayed

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

LLM agents: curated papers on planning, memory, tool use, multi-agent systems, evaluation, and safety.

AI

README.md

Original

🤖 Awesome LLM Agent Papers

English · 한국어 · 简体中文 · 日本語 · Português · Español

200+ must-read papers, updated regularly: the annotated reading list for building
LLM agents that plan, remember, use tools, and cooperate. Companion to the survey “LLM Agents: A Survey.”

Awesome 200+ curated papers DOI: 10.20944/preprints202608.0265.v1 License: MIT PRs welcome

GitHub stars GitHub forks

📄 Read the survey → “LLM Agents: A Survey”  ·  PDF in this repo
📖 Online book on GitBook  ·  ⭐ Start with the Starter Kit

LLM agents · LLM agent papers · autonomous agents · agentic AI · multi-agent systems · tool use · ReAct · planning · memory · agent benchmarks · agent safety & prompt injection

Taxonomy of LLM-agent research

✨ Highlights

What's here
📚 A superset of the survey The 228 references behind “LLM Agents: A Survey,” plus papers that came out after the paper was written. New papers are still being added.
🧭 Organized by function 10 sections following the survey's structure: Surveys, Architectures, Planning, Memory, Tool Use, Multi-Agent, Environments, Applications, Evaluation, Safety.
✍️ Annotated Each entry has a one-line note on what it contributes, its venue and year, and a [code] link where an official implementation exists.
⭐ Starter Kit A 10-paper list for getting oriented, with a note on why each one is worth reading first.
🔎 Navigable A Contents index with per-section counts and collapsible sections.

Topics covered: cognitive architectures · ReAct & reasoning-and-acting · long-horizon planning · agent memory · tool-augmented LLMs · multi-agent collaboration · web / code / embodied agents · agent benchmarks & evaluation · safety, alignment & indirect prompt injection.

📖 The survey as an online book: LLM Agents: A Survey on GitBook rewrites the paper for readers new to the area, in ten chapters that start from the basic concepts. As in the paper, approaches that fell short get as much space as the ones that worked.

🔁 The list for the follow-up survey: Awesome Agent Loop Papers collects work on the agent loop itself, with 524 papers and 60 open-source projects (frameworks, coding harnesses, memory and sandbox infrastructure, skill libraries, registries). It accompanies The Agent Loop: A Survey of Control Strategies, Skills, and Harnesses for LLM Agents.

This repository collects must-read papers on LLM-based agents: language models equipped with planning, memory, tool use, and multi-agent coordination to pursue goals over long horizons. Papers follow the taxonomy of the accompanying survey, covering the core components of an agent, the environments and applications they are deployed in, and the cross-cutting concerns of evaluation and safety. Each entry links to the paper and, where an official implementation exists, to its code.

This is a curated, continuously updated selection rather than a copy of the survey's bibliography. Sections are collapsed by default; click Show N papers to expand.

Legend: ⭐ = Starter Kit pick (read these first) · [code] = link to an official implementation.

⭐ Starter Kit

New to the area? These ten papers are a good place to start.

# Paper Area Why start here
1 ReAct: Synergizing Reasoning and Acting Planning The template for the modern agent loop: interleave reasoning with actions.
2 Reflexion: Verbal Reinforcement Learning Planning Self-reflection stored in memory as a gradient-free improvement loop.
3 Toolformer: LMs Can Teach Themselves to Use Tools Tool Use The seminal self-supervised tool-use paper.
4 Generative Agents: Interactive Simulacra Multi-Agent Memory + reflection at population scale; the canonical agent-memory design.
5 Voyager: An Open-Ended Embodied Agent Memory / Env. Lifelong learning via a growing library of executable skills.
6 Cognitive Architectures for Language Agents (CoALA) Foundations The vocabulary (memory, action space, decision loop) this list is organized around.
7 A Survey on LLM-based Autonomous Agents Survey The canonical general survey of the field.
8 LLM-based Multi-Agents: A Survey Multi-Agent The standard reference for the multi-agent branch.
9 AgentBench: Evaluating LLMs as Agents Evaluation The standard cross-environment agent benchmark.
10 Not what you've signed up for (Indirect Prompt Injection) Safety The founding paper of the agent-security threat model.

🔥 10 to Watch (2026)

Fresh 2026 work that is already drawing attention.

Paper Area Stars
GenericAgent: A Token-Efficient Self-Evolving LLM Agent via Contextual Information Density Maximization Architectures stars
SimpleMem: Efficient Lifelong Memory for LLM Agents Memory stars
AutoSci: A Memory-Centric Agentic System for the Full Scientific Research Lifecycle Applications stars
Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents Environments stars
AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security Safety stars
Agentic Reasoning for Large Language Models Surveys stars
UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents Tool Use stars
Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration Multi-Agent stars
Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents (DataAgentBench) Evaluation stars
Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents Planning stars

↑ Back to Contents

Contents

🧭 Background

📚 Surveys & Position Papers (62)

Corresponds to §1-§3 (Introduction, Background, Taxonomy).

Show 62 papers

↑ Back to Contents

🏗️ Agent Architectures & Frameworks (54)

Corresponds to §2 (Background) and the running examples throughout.

Show 54 papers

↑ Back to Contents

🧱 Part I: Core Components

🧠 Planning & Reasoning (55)

Corresponds to §4 (Planning and Reasoning).

Show 55 papers

↑ Back to Contents

💾 Memory (60)

Corresponds to §5 (Memory).

Show 60 papers

↑ Back to Contents

🔧 Tool Use (51)

Corresponds to §6 (Tool Use and Action Execution).

Show 51 papers

↑ Back to Contents

🤝 Multi-Agent Systems (57)

Corresponds to §7 (Multi-Agent Systems).

Show 57 papers

↑ Back to Contents

🌍 Part II: Agents in Context

🌐 Interactive Environments (60)

Corresponds to §8 (Agents in Interactive Environments).

Show 60 papers

↑ Back to Contents

🚀 Applications (60)

Corresponds to §10 (Applications).

Show 60 papers

↑ Back to Contents

⚖️ Part III: Cross-Cutting Concerns

📊 Evaluation & Benchmarks (62)

Corresponds to §9 (Evaluation and Benchmarks).

Show 62 papers
  • GAIA: a benchmark for General AI Assistants (Mialon et al., ICLR 2024) - Reference benchmark for generalist, tool-using agent assistants; underlies popular public leaderboards tracking frontier agent progress.

  • SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (Jimenez et al., ICLR 2024) - De facto standard benchmark for coding/software-engineering agents; spawned the SWE-bench Verified/Lite/Live/Multimodal family. [code]

  • MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation (Huang et al., arXiv 2023) - Seminal benchmark for the 'AI research agent'/'ML engineering agent' evaluation subfield, precursor to MLE-bench, RE-Bench, etc. [code]

  • τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao et al., arXiv 2024) - Pioneered evaluation of agent-user interaction and policy compliance; standard reference for enterprise/customer-service agent evaluation. [code]

  • AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents (Ma et al., NeurIPS 2024) - Widely-used unified evaluation toolkit addressing the coarse pass/fail scoring gap, directly relevant to evaluation-methodology framing. [code]

  • SmartPlay: A Benchmark for LLMs as Intelligent Agents (Wu et al., ICLR 2024) - Capability-decomposed evaluation methodology that influenced later fine-grained agent capability benchmarks. [code]

  • TravelPlanner: A Benchmark for Real-World Planning with Language Agents (Xie et al., ICML 2024) - Widely-cited stress test for complex constrained multi-tool planning, illustrating agents' distance from reliable long-horizon planning. [code]

  • InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback (Yang et al., NeurIPS 2023) - Established the interactive, execution-feedback evaluation paradigm underlying later coding-agent and terminal-agent benchmarks. [code]

  • GTA: A Benchmark for General Tool Agents (Wang et al., NeurIPS 2024) - Addresses realism gaps (implicit intent, authentic multimodal context) left by earlier synthetic tool-use benchmarks. [code]

  • Survey on Evaluation of LLM-based Agents (Yehudai et al., arXiv 2025) - Directly on-topic survey providing a ready-made taxonomy for a new agent survey's evaluation section.

  • Evaluation and Benchmarking of LLM Agents: A Survey (Mohammadi et al., KDD 2025) - Independently-developed sub-topic survey complementing Yehudai et al., useful for cross-checking taxonomy coverage.

  • τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment (Barres et al., arXiv 2025) - Extends τ-bench to settings where user and agent both act on the environment. [code]

  • Agent-as-a-Judge: Evaluate Agents with Agents (Zhuge et al., arXiv 2024) - Systematizes agentic evaluation of agents; the reference point for judge-circularity concerns. [code]

  • Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation (Kapoor et al., arXiv 2025) - Standardized harness that re-evaluates agents at scale with cost reported alongside accuracy.

  • Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reports (Yao et al., arXiv 2025) - Evaluates deep-research agents from answers to reports across semantic quality, topical focus, and retrieval trustworthiness. [code]

  • Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces (Merrill et al., arXiv 2026) - Hard, realistic command-line tasks; a de facto standard for terminal agents. [code]

  • Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents (DataAgentBench) (Ma et al., arXiv 2026) - Benchmark across heterogeneous database systems; frontier models reach only 38% accuracy on complex data questions. [code]

  • AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts (Li et al., arXiv 2026) - 32 real-world long-horizon scenarios testing autonomous agents under up to 1M-token contexts. [code]

  • Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation (Pysklo et al., arXiv 2026) - Benchmarks agents on enterprise API tasks via sandboxed code execution with state-diff (not trace-matching) success criteria. [code]

  • When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents (ToolMaze) (Zhu et al., arXiv 2026) - DAG-based benchmark with a perturbation taxonomy testing agent replanning and recovery when tool calls fail. [code]

  • Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values (Dong et al., arXiv 2026) - First dedicated agent-values benchmark: 394 executable environments and 4,335 value-conflict tasks across 28 value systems.

  • How Many Tasks Are Enough for Agent Benchmark Decisions? A Replay Analysis of Public LLM Agent Benchmarks (Huang et al., arXiv 2026) - Replay analysis asking how many tasks a benchmark actually needs before its ranking is trustworthy. [code]

  • Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents (Patel et al., arXiv 2026) - Argues agent leaderboards need predictive validity, not just a score, to say anything about deployment.

  • AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments (Xi et al., arXiv 2026) - Benchmarks agents in de-idealized environments, closing the gap between clean benchmark worlds and real ones. [code]

  • Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents (Yi et al., arXiv 2026) - Measures how much the harness alone shifts an agent's beliefs across steps, isolating a confound in agent evaluation. [code]

  • Rethinking the Evaluation of Harness Evolution for Agents (Wang et al., arXiv 2026) - Rethinks how harness evolution should be evaluated, given that the harness moves the number as much as the model. [code]

  • ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions (Gupta et al., arXiv 2026) - Benchmarks agent reliability under production-like stress rather than single clean runs.

  • AgentAtlas: Beyond Outcome Leaderboards for LLM Agents (Mazaheri et al., arXiv 2026) - Moves agent evaluation beyond outcome leaderboards toward per-decision diagnosis of where control breaks.

  • CUBE: A Standard for Unifying Agent Benchmarks (Lacoste et al., arXiv 2026) - Proposes a standard for unifying heterogeneous agent benchmarks under one interface. [code]

  • UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (Chen et al., arXiv 2026) - Introduces UniClawBench, a capability-driven bilingual benchmark of 400 tasks executed in live Docker environments with step-wise checkpoints, evaluating proactive LLM agents across skill usage, exploration, long-context reasoning, multimodal understanding and cross-platform coordination, with a hidden supervisor agent so that multi-turn feedback never leaks the grading criteria. [code]

  • PolyWorkBench: Benchmarking LLM Agents for Cross-Lingual Long-Horizon Workflows (Li et al., arXiv 2026) - Introduces PolyWorkBench, a benchmark of 67 tasks spanning commerce, knowledge work, legal analysis, localization, and manufacturing that evaluates LLM agents on multilingual long-horizon workplace workflows via a hybrid of rubric-based structural scoring, executable state checks and an LLM judge; performance varies widely across languages and drops sharply on the harder cross-lingual tasks.

  • MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations (Shi et al., arXiv 2026) - Gives an agent an online store for 365 simulated days, built on 98,843 real product records from the 1688 wholesale marketplace, with 26 tools, refunds and bad reviews that surface days after the order, and final net assets as the score: across 48 runs of eight LLMs under a minimal ReAct loop and the Hermes agent framework, the best setup, Qwen3.7-Max with Hermes, ends with 27.3% of the mean net assets of three human participants with no e-commerce experience, a rule-based script that checks the store once a day beats 6 of the 16 setups, and Kimi K2.6 under Hermes decides on day 104 that the store cannot recover and then takes no store action in 355 of its remaining 523 decision windows; the released code runs on a synthetic catalog of 1,000 products because the real records and daily market reports are not redistributed, so the paper's numbers cannot be reproduced from the release alone. [code]

  • EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer (Gao et al., arXiv 2026) - Introduces a benchmark that evaluates LLM agent self-evolution as procedural-knowledge transfer, extracting trace-grounded "Abilities" from agent executions and organizing them into domain-specific Ability Graphs across web research, algorithmic reasoning, software engineering and knowledge work; curated Abilities transfer reliably across model families, but no automatic method gains in every setting.

  • Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle (Wang et al., arXiv 2026) - Introduces AARRI-Bench, a containerized benchmark suite evaluating whether frontier LLMs and agentic harnesses can perform entry-level research-intern tasks across the research lifecycle. [code]

  • ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences (Nguyen et al., arXiv 2026) - Introduces ReplicatorBench, a benchmark of human-verified replicable and non-replicable claims for evaluating LLM agents on replicating social and behavioral science research across data retrieval, experiment design/execution, and interpretation of results; agents design and run the experiments well but struggle to retrieve the new data a replication needs. [code]

  • Benchmark Test-Time Scaling of General LLM Agents (Li et al., arXiv 2026) - Introduces General AgentBench, a unified benchmark for evaluating general-purpose LLM agents across search, coding, reasoning, and tool-use, and finds that sequential and parallel test-time scaling fail to improve performance due to a context ceiling in the sequential case and a verification gap in the parallel one, while ten leading agents degrade substantially when moved from domain-specific evaluations into the general setting. [code]

  • BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks (Tu et al., arXiv 2026) - Turns frontier models on the benchmarks themselves, finding 12 author-confirmed issues in ScienceAgentBench including errors that made tasks unsolvable, and matching 83.3% of expert-identified issues on BIXBench at under USD 15 per 50-task audit.

  • Automated Benchmark Auditing for AI Agents and Large Language Models (Wang et al., arXiv 2026) - Introduces Auto Benchmark Audit (ABA), an agentic framework that audited 168 benchmarks across nine domains and found critical issues (ambiguous design, execution conflicts, incorrect ground truths) in over 25.7% of tasks; filtering those tasks out reshuffles model rankings and raises average scores on SWE-bench Verified and Terminal-Bench 2 by 9.9% and 9.6%.

  • PerspectiveGap: A Benchmark for Multi-Agent Orchestration Prompting (Sun et al., arXiv 2026) - Isolates orchestration prompt writing as its own capability: 110 scenarios over 10 topologies on which 33 models average a 17.2% pass rate. [code]

  • ClawBench: Can AI Agents Complete Everyday Online Tasks? (Zhang et al., arXiv 2026) - Puts browser agents on 153 everyday tasks across 144 live production sites, intercepting the final request so nothing is actually bought or booked; the strongest model tested clears a third of them. [code]

  • Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI (Shao et al., arXiv 2026) - Audits 2,385 traces across 15 agent benchmarks and finds exposures and reward hacking in about two thirds of Frontier Science and AutoLab tasks, inflating scores by 0.45 to 1.00.

  • The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation (Yu et al., arXiv 2026) - Agent configurations with identical accuracy differ 15.7x in bytes left on disk, so persistent storage belongs in the report alongside accuracy and reconstructability.

  • OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding (Zhou et al., arXiv 2026) - Prices 100 long-horizon office tasks by the human labor they replace, averaging 2.32 hours, and finds frontier models far cheaper than workers yet well short of their deliverable quality. [code]

  • AV-AIVAT: 74x Cheaper Agent Evaluation with Certified Anytime-Valid Stopping in Imperfect-Information Games (Li et al., arXiv 2026) - Lets an agent comparison stop as soon as the evidence settles without voiding the stated confidence level, pairing variance reduction with continuously monitored confidence sequences; over 71,439 paired poker hands, raw outcomes need a median 74x as many games.

  • PATH-Bench: Path-Dependent Evaluation of Lifelong Agents (Yang et al., arXiv 2026) - Evaluates lifelong agents on the order of their experience rather than task by task, measuring forward transfer, backward transfer and forgetting, and finds that strong transfer does not imply retention and that later experience can undo earlier gains.

  • Benchmarking LLM Judges for Mobile Agent Evaluation (Wang et al., arXiv 2026) - Tests six LLM-as-judge methods against 931 human-annotated mobile agent trajectories and finds a simple sampled-screenshot baseline competitive with or better than purpose-built judges, with the backbone model, not the pipeline, driving judge quality, and two backends showing opposite conservative and permissive failure profiles.

  • OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agents (Caciolai et al., arXiv 2026) - Expanding GAIA2 into ten languages across five writing systems exposes a cross-lingual gap of 8.8 to 18.4 pass@3 points that concentrates on tool orchestration rather than quantitative reasoning and does not close with model scale, with error attribution assigning 55% of it to the model and bounding translation contamination at 6.4% of scenario-language pairs.

  • LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering (Wang et al., arXiv 2026) - Scores the model that steers rather than the model that codes: after each coding round the Controller under evaluation reads a structured run summary and tells a separate fixed Worker what to do next, what to verify, or whether to stop, across three settings trading execution scope against cost; the best strict success rate on full tasks is 24.69 per cent, and the cheap setting that never runs the Worker orders Controllers almost identically to the expensive one at a Spearman correlation of 0.97. [code]

  • τ^τ-Bench: An Environment for End-To-End, Realistic Agent Construction (Shi et al., arXiv 2026) - Makes building the agent the task: a developer agent inherits a codebase, a production API, a client holding the requirements and a cap on serving spend, then ships a customer-service agent that is scored by deploying it against held-out simulated users, and across 53 tasks the strongest configuration passes 23.9 per cent of evaluation simulations against an expert-authored ceiling of 82.2 per cent, failing in recognizably human ways by querying the records shallowly, telling the client almost nothing and shipping the first architecture that runs.

  • Autonomous Evaluation and Refinement of Digital Agents (Pan et al., COLM 2024) - Builds agent evaluators at several cost points that agree with oracle metrics 74.4 to 92.9% of the time, then uses them as rewards to improve the state of the art on WebArena by 29% without extra supervision. [code]

  • JEV-as-a-Judge: Accept When Confident, Escalate When Unsure (Li et al., arXiv 2026) - A judge that returns a decision rather than text comes within three points of the strongest LLM judge on ordinary preference and factuality at 0.36% of its fee, falls further behind when a derivation must be checked or a wrong answer is well written, and an accept-or-escalate cascade keeps 99% of the stronger judge's accuracy.

  • TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces (Min et al., arXiv 2026) - Builds 107 benchmarks from 252,557 Claude Code and OpenClaw deployment sessions by cutting each trace where the original agent showed an unwanted behavior and grading a model's next turn against a behavior-specific rubric, with no environment replay: nine frontier LLMs pass 26.7% on average, 67.9% when the rubric asks only for a well-formed tool call but 8.1% when it asks for a required check before continuing, and responses that open with a TodoWrite planning call pass 4.6% against 30.8% for other responses on the same instances, which the authors report as an association; the traces are not released, so the benchmarks cannot be rebuilt from public data.

  • TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution (Yao et al., arXiv 2026) - Audits coding-benchmark trials that pass the official tests: a GPT-5.5 auditor writes tests for each trial aimed at requirements in the prompt that the patch may break, and keeps only tests the reference patch passes, so every refutation comes with a test that can be replayed; across 6 model backends and 5 benchmarks it refutes 34.4% of the trials judged correct, lowering the resolution rate from 50.6% to 33.2%, with refutation rates from 7.6% on SWE-bench Verified to 53.6% on SWE-Marathon, though the auditor and its reviewer are LLMs, so a refutation is only as good as their reading of the prompt.

  • AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks (Lù et al., arXiv 2026) - Tests LLM judges of long computer-use tasks on 1,373 instruction and trajectory pairs from 166 hours of human recordings on three operating systems, where each negative pairs a recording with the instruction written for a closely related task, and runs eleven judges both on the full trajectory and as coding agents in five harnesses: GPT-5.5 in Codex is best at 80.9% balanced accuracy against 68.6% when prompted directly, while Qwen 3.6 27B scores 7.2 points lower inside OpenCode than without it and, given no instruction, calls 98.9% of trajectories successful; the negatives are near-miss swaps rather than real agent failures. [code]

  • Finding the Right Fit: Model-Harness Interactions across Agent Tasks (Li et al., arXiv 2026) - Pairs five models with four configurable harnesses, OpenHands, DeepSeek Harness, PI and openJiuwen, on TUA-Bench, ALE-CLI and Terminal-Bench 4, and adds the native Codex and Claude Code pairings, for 66 configurations: model rankings reverse across harnesses, with Claude 7.94 points ahead of GPT in OpenHands on Terminal-Bench 4 but 30.16 points behind in PI, the best harness changes between benchmarks for four of the five models, a model's own vendor harness is not reliably its best, and GPT-6 Astra scores 60.32% in PI at $4.66 per task against 52.38% in DeepSeek Harness at $19.94; each task counts once, from its last run. [code]

  • Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym (Kaesberg et al., arXiv 2026) - Turns 2D grid pathfinding with seven kinds of interacting rule cells into a Gymnasium task in which the agent draws the path one move at a time from the legal moves it is offered, optionally with backtracking, and tests eight open-weight models on 500 puzzles: the best, GPT-OSS 120B, solves 16.0% against 98.0% for people and 6.4% for a shortest-path agent that ignores the rules, the step-by-step format adds up to 5.4 points over one-shot answers for weaker models by removing formatting errors but costs stronger ones up to 5.6, backtracking raises GPT-OSS's completion rate from 85% to 94% while its solve rate falls 5.8 points, and Qwen3-VL-32B given the puzzle as an image solves 2.8% against 10.2% from text; no proprietary model is tested. [code]

  • Super Research: Answering Highly Complex Questions with Large Language Models through Super Deep and Super Wide Research (Dong et al., arXiv 2026) - Asks 300 open research questions in ten domains, drafted with GPT-4o and Gemini Deep Research and then vetted by domain experts, and scores each report against a reference graph of facts and insights that agents built and experts refined, on coverage, logical consistency, a quiz an LLM answers from the report alone, objectivity and citation spread: Gemini Deep Research leads 12 systems at 28.62 overall, Kimi-k2 answering with its built-in search (26.16) beats o3 Deep Research (22.04), and when 1 to 3 facts are removed from or added to a report the graph-based scores move the right way in 57.4% to 79.6% of cases against 14.8% to 22.2% for an LLM judge; the release has the questions and 3,600 reports but neither the reference graphs nor a scorer, so a new agent cannot yet be scored the way the paper does. [code]

  • SciΠvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents (Sun et al., arXiv 2026) - Builds 248 two-way choices from six months in which senior researchers ran LLM agents on 26 projects in six fields and corrected their research decisions: given the project record from just before a correction, a model picks between the agent's original plan and the corrected one. At medium reasoning effort four of five frontier models pick the original plan more often than not, Claude Opus 5 at 38.9% and GPT-6 Astra, the exception, at 59.5%; the best score in any setting is 63.5%, against 73.4% for 13 PhD students and postdocs new to the projects who had an agent assistant, and at that effort every model scores lower with the project record than with the two plans alone. The cases were kept because those same five models split on them, so the scores show where models go wrong rather than how often, and the questions, prompts and project records are released, without the models' response logs. [code]

  • Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution (Huang et al., arXiv 2026) - Builds 128 executable digital-workspace tasks from six agent benchmarks, marks 3,385 sensitive items in 100 of them, some already in the workspaces and some inserted by an LLM, and scores every run on completion, cloud cost and the share of those items the cloud model gets to see, comparing four ways of splitting the work between Qwen edge models and GLM-5.3 in the cloud. With Qwen3.5-2B, routing each step by the edge model's token entropy lifts completion from 12.75 to 39.97 at 22.7% of cloud-only cost, while letting the edge model call the cloud itself adds only 2 to 3 points; with Qwen3.5-9B, cloud advice that the edge model asks for reaches 60.46 against 82.70 for cloud-only at about 15% of its cost but shows the cloud 45.46% of the sensitive items; with Qwen3.7-Plus the four ways differ by 3.67 points in completion and by 66.23 in exposure. The tasks were filtered with runs of the same 2B and 9B edge models and GLM-5.3, dropping those both ends solved and those neither made headway on, nothing is redacted before a cloud call, and the release covers three of the four ways, without the delegation mode or the run logs. [code]

  • Measuring AI Agents as Teammates with Collaborative Intelligence (Wu et al., arXiv 2026) - Puts the model under test in a team chat of 2 to 5 with live LLM partners and one scripted teammate who creates the problem, over 240 tasks in eight dimensions of teamwork built from accident reports, clinical cases, negotiation datasets and coding benchmarks, and scores each model only on the tasks it solves alone. Every one of 18 models scores 20.2 to 46.6 points lower in the test than in a team where nothing goes wrong, and the best is 82.2, Gemini 4 Argon at high reasoning effort; frontier models pass at least 93% of the tests on refusing unsafe shortcuts and on reporting misconduct, but in a conflict they ask what the pushy teammate needs in 99% of runs and then accept its weaker split in 81%, and they pass 81% of tasks taken over from a weaker model that had cheated, against 97% when they start those tasks themselves. The scripted teammate speaks fixed lines and branches on how LLM judges label each reply, the live partners are LLMs rather than people, the alone and baseline versions are run once each, and 80 of the 240 tasks are held out and scored by the authors on request. [code]

  • TeamGym: A Gym for Team Decision-Making by AI Agents with Distributed Information (Park et al., arXiv 2026) - Builds team decision problems from 12 legal, finance, medical and software-engineering corpora, finds for each the smallest set of facts that together settle the reference decision while each one is needed, and controls which participant starts with which fact. Averaged over eight models, splitting those facts across participants lowers success from 63.4% for one model holding them all to 40.5%, and making participants find them in their own source documents lowers it to 20.6%; GRPO training of Qwen2.5-7B, with a GPT-5.5 judge scoring only the final decision, raises FinDVer validation from 21.85% to 41.08% and the share of episodes in which every participant speaks from 61.7% to 86.7%, which the reward does not count. The facts were picked and checked with models, several decisions are graded by an LLM judge, each training result comes from one run, gains on CUAD barely carried over to MAUD, and the code is not yet released.

  • TeamBench: Evaluating Agent Coordination under Enforced Role Separation (Kim et al., arXiv 2026) - Runs Planner, Executor and Verifier in separate containers so that no role can read the full requirements, edit the workspace and certify the result all at once, over 851 task templates, 650 of them adapted from GitHub bug reports, each with a deterministic grader. On its 90-task leaderboard the best of 13 models, Claude Opus 4.7, passes 37.8% as a team against 35.6% alone; with Gemini-3 Flash in every role on 155 tasks, the team beats a single agent by 15.7 points on the fifth of tasks the single agent finds hardest but trails it by 6.8 to 10.1 on the three fifths it finds easiest, and dropping the Verifier raises the mean partial score by 5.5. Across the role-mixing runs, Verifiers that return a verdict approve 49.4% of the submissions the grader rejects, 22.3% if runs without a verdict are counted too, and prompt-only roles reach a pass rate statistically indistinguishable from enforced ones while the Verifier tries to edit the Executor's code 3.6 times as often. Each role works in one file-based pass with capped turns and no multi-round dialogue between roles, and open-weight models of 30B or less fail mostly on tool calls, so for them the benchmark measures tool-call reliability more than coordination. [code]

↑ Back to Contents

🛡️ Safety & Alignment (62)

Corresponds to §11 (Safety, Security, and Trustworthiness).

Show 62 papers

↑ Back to Contents

🔗 Related Awesome Lists

Other useful reading lists in the same space:

Maintain a related list? Open a PR to add it here; we're happy to cross-link.

↑ Back to Contents

📄 Citation

The survey is posted on Preprints.org as LLM Agents: A Survey, DOI 10.20944/preprints202608.0265.v1. That versioned record is the one to cite. The same 47-page paper is also in this repository at paper/llm-agents-a-survey.pdf so you can read it without leaving GitHub.

If this list or the survey is useful to you, please cite:

@article{lee2026llmagents,
  title     = {LLM Agents: A Survey},
  author    = {Lee, Jungseob},
  year      = {2026},
  month     = {August},
  journal   = {Preprints},
  publisher = {Preprints},
  doi       = {10.20944/preprints202608.0265.v1},
  url       = {https://doi.org/10.20944/preprints202608.0265.v1}
}

GitHub's Cite this repository button reads CITATION.cff and returns the same record in APA or BibTeX.

The follow-up survey, The Agent Loop: A Survey of Control Strategies, Skills, and Harnesses for LLM Agents, has its own record: DOI 10.2139/ssrn.7186738. Cite whichever one you actually drew on.

🤝 Contributing

The agent literature is arriving faster than any one person can track, on the order of a thousand new papers a month. I read a lot for this list, but I have certainly missed strong papers and methods. If one belongs here (including your own), please help:

  • Open a PR adding it to the right section with a verifiable link and a one-line why it matters (and a [code] link if an implementation exists), or
  • Open an issue with the link and I'll triage quickly.

Corrections, sharper annotations, and whole new sections are just as welcome. See CONTRIBUTING.md for the entry format.

Most of the languages this list is translated into aren't my first language. If a sentence in one of the translations reads oddly or a term is translated wrong, please open an issue, or fix it directly with a PR against that file.

👥 Contributors

This list is community-maintained. Thanks to everyone who has suggested, verified, or annotated a paper:

Show 24 contributors
Contributor Contributed
@ybkim95 @ybkim95 The Collaborative Intelligence Benchmark (a model as a teammate, scored only on tasks it solves alone), TeamGym (team decisions with the deciding facts split across participants) and TeamBench (Planner, Executor and Verifier kept apart by container permissions), in Evaluation & Benchmarks (#34). Submitted by one of their authors, the first author of TeamBench
@passionate11 @passionate11 EC2-Bench, 128 workspace tasks that score edge-cloud collaboration on completion, cloud cost and privacy exposure, submitted by its first author, in Evaluation & Benchmarks (#33); ToolMaster, which trains a model to try tools out before relying on them, submitted by one of its two first authors, in Tool Use (#28)
@ChaoYue0307 @ChaoYue0307 Harness Engineering for Language Agents, a position paper that splits the harness into control, agency and runtime and proposes HarnessCard for reporting it, submitted by its first author, in Surveys & Position Papers (#27)
@WhymustIhaveaname @WhymustIhaveaname AgonAlpha, alpha mining whose reviewer reruns each candidate and can veto it, and AutoSR, symbolic regression run as a search over research states, both submitted by one of their co-first authors, in Applications (#32); SciΠvot, a benchmark of research decisions built from senior researchers' corrections of LLM agents, submitted by its first author, in Evaluation & Benchmarks (#29); 3 research-agent and orchestration papers: AutoNumerics, OptimAI, PerspectiveGap (#1); the Agon code link, and the report that led to nine truncated glosses being rewritten (#5)
@dolev31 @dolev31 TabAgent, which replaces an agent's repeated tool-shortlisting call with a small classifier trained on its own runs, submitted by its first author, in Tool Use (#30)
@segevshlomovIBM @segevshlomovIBM From Benchmarks to Business Impact, IBM's pilot of its CUGA agent in talent acquisition, submitted by its first author, in Applications (#25)
@KhanCold @KhanCold MerchantBench, a benchmark that has an agent run an online store for 365 simulated days, submitted by its first author, in Evaluation & Benchmarks (#24)
@cnsdqd-dyb @cnsdqd-dyb Super Research, a benchmark of 300 open research questions with reports from 12 systems, submitted by its first author, in Evaluation & Benchmarks (#23)
@lkaesberg @lkaesberg Voting or Consensus? (decision protocols in debate), Stay Focused (problem drift in debate) and MALLM, the debate framework both ran on, in Multi-Agent Systems; Spatial-Gym, step-by-step spatial reasoning for agents, in Evaluation & Benchmarks (#22). Submitted by one of their authors
@markovchain-builder @markovchain-builder SELF, agent training from environment feedback without rewards, in Planning & Reasoning; EFRE, a repertoire of prompts for continual learning with a frozen LLM, in Memory (#21)
@ZhishanQ @ZhishanQ TraceDance, benchmarks built from real agent deployment traces, submitted by its first author, in Evaluation & Benchmarks (#20)
@Gavin-M34 @Gavin-M34 Jev in the Wild, a study of how public GitHub projects use Jev, in Surveys & Position Papers (#19)
@shubhamrgandhi @shubhamrgandhi Steer, Don't Solve, a small critic model that steers a larger coding agent, in Planning & Reasoning (#16); DRACO, rubric-based credit assignment for agents trained without a verifier, in Tool Use (#17). Submitted by their first author
@sunyuhan19981208 @sunyuhan19981208 The TaoLive report on training a compact model to keep working as its harness changes, submitted by one of its authors, in Applications (#15)
@burgerseater @burgerseater LoopArena, a benchmark that scores the model steering a coding agent rather than the one writing the code, suggested by one of its authors, in Evaluation & Benchmarks (#14)
@dukesun99 @dukesun99 Corpus2Skill in Memory, OrchMAS in Multi-Agent Systems, and an IR-for-agents position paper in Surveys & Position Papers, submitted by an author on all three (#13)
@zhongzero @zhongzero ForeDreamer, a dual-agent memory architecture for forecasting, submitted by one of its authors, in Memory (#12)
@Nicolas99-9 @Nicolas99-9 CityReal, a large-scale human-aligned urban simulation, in Multi-Agent Systems (#10)
@BobbyZhouZijian @BobbyZhouZijian CORAL, an autonomous multi-agent evolution framework, submitted by one of its authors, in Agent Architectures & Frameworks (#9)
@razzant @razzant Ouroboros, a self-developing coding-agent harness, submitted by its maintainer, in Agent Architectures & Frameworks (#8)
@reacher-z @reacher-z Dr. Bench, a deep-research-agent evaluation, submitted by one of its authors, in Evaluation & Benchmarks (#7); ClawBench, a live-web browser-agent benchmark, same section (#3)
@JEONGSEJIN @JEONGSEJIN WebAgent and the world-model-augmented web agent, in Interactive Environments (#6)
@jinmang2 @jinmang2 6 agent-memory systems: MemoryOS, Zep, Nemori, MemOS, G-Memory, ACE (#2)
@js-lee-AI @js-lee-AI Maintainer

Listed by hand rather than generated, so that it credits the people who contributed, not whoever happened to run git commit. See the contributor graph for commit-level history.

Want your avatar here? See CONTRIBUTING.md: a single well-formatted PR is all it takes.

📜 License

Released under the MIT License.

🗓️ Updates

  • 2026-11-06: The Collaborative Intelligence Benchmark, TeamGym and TeamBench added to Evaluation & Benchmarks by one of their authors (@ybkim95, #34). In TeamGym, splitting the deciding facts across a team instead of giving them all to one model lowers success from 63.4% to 40.5%. 580 to 583.
  • 2026-11-05: EC2-Bench added to Evaluation & Benchmarks by its first author (@passionate11, #33). With Qwen3.7-Plus on the edge, the four ways of bringing in the cloud model differ by 3.67 points in completion and by 66.23 in how much of the sensitive data the cloud sees. 579 to 580.
  • 2026-11-02: Harness Engineering for Language Agents added to Surveys & Position Papers by its first author (@ChaoYue0307, #27). In its audit of 63 works, the 38 papers and benchmarks most often foreground interfaces and observability, and the 25 engineering notes foreground runtime, control and governance. 578 to 579.
  • 2026-11-02: AgonAlpha and AutoSR added to Applications by one of their co-first authors (@WhymustIhaveaname, #32). AutoSR finds the exact equation on three cp3-bench problems that none of 12 published symbolic-regression systems solved, from one successful run per problem. 576 to 578.
  • 2026-10-28: TabAgent added to Tool Use by its first author (@dolev31, #30). A classifier trained on the agent's own runs takes over GPT-5.4's tool shortlisting on AppWorld at about 3 ms a call instead of 7.5 s, and a live agent's task success moves from 29.2% to 27.1%, a gap too small to tell apart on 24 tasks. 575 to 576.
  • 2026-10-28: SciΠvot added to Evaluation & Benchmarks by its first author (@WhymustIhaveaname, #29). Asked to choose between an agent's research plan and the one a senior researcher corrected it to, the best frontier model reaches 63.5%, against 73.4% for researchers new to the projects. 574 to 575.
  • 2026-10-28: ToolMaster added to Tool Use by one of its two first authors (@passionate11, #28). Trained to try tools before relying on them, Qwen2.5-7B reaches 66.4 on StableToolBench, against 60.2 for GPT-4o without training. 573 to 574.
  • 2026-10-19: From Benchmarks to Business Impact added to Applications by its first author (@segevshlomovIBM, #25). In a talent-acquisition pilot that is not yet in full production, IBM's CUGA agent reaches 87% on the 26 tasks of the new BPO-TA set. 572 to 573.
  • 2026-10-17: MerchantBench added to Evaluation & Benchmarks by its first author (@KhanCold, #24). After 365 simulated days of running an online store, the best of 16 LLM setups ends with 27.3% of the human participants' mean net assets. 571 to 572.
  • 2026-10-16: Super Research added to Evaluation & Benchmarks by its first author (@cnsdqd-dyb, #23). On its 300 research questions the best of 12 systems, Gemini Deep Research, scores 28.62 out of 100. 570 to 571.
  • 2026-10-15: Voting or Consensus?, Stay Focused and MALLM added to Multi-Agent Systems and Spatial-Gym to Evaluation & Benchmarks by one of their authors (@lkaesberg, #22). In Spatial-Gym the best of eight open-weight models solves 16.0% of the grid puzzles, against 98.0% for people. 566 to 570.
  • 2026-10-14: +30 papers posted since mid-September 2026, three per section. Several report negative results; in one, 34.4 per cent of the coding-agent runs that benchmarks judge correct break a requirement stated in the task. 536 to 566.
  • 2026-10-12: SELF added to Planning & Reasoning and EFRE to Memory (@markovchain-builder, #21). SELF trains agents without rewards, and with Qwen3-8B it reaches 43.2% on τ-bench Retail against 39.1% for GRPO trained on rewards. 534 to 536.
  • 2026-10-03: TraceDance added to Evaluation & Benchmarks by its first author (@ZhishanQ, #20). It builds tests from real agent sessions, and on them nine frontier LLMs pass 8.1% of the cases that call for a check before continuing. 533 to 534.
  • 2026-09-30: Jev in the Wild, a study of how 2,170 public GitHub projects use Jev, added to Surveys & Position Papers (@Gavin-M34, #19). 532 to 533.
  • 2026-09-24: The list and the online book on GitBook are now also available in Brazilian Portuguese and Spanish.
  • 2026-09-24: The survey is now on GitBook as an online book, written for readers new to the area.
  • 2026-09-24: The list is now available in Korean, Simplified Chinese and Japanese; choose a language at the top. The notes and prose are translated, and paper titles and terms of art are left in English. While translating I found notes that stopped mid-sentence and entries missing their first author, and fixed them.
  • 2026-09-23: Twelve papers on models inside the agent loop that answer with a typed decision instead of text and hand off to an LLM when they aren't sure. Jev's release drew wide attention to the idea, and four of the papers study Jev itself; in one of them, Jev follows an option's name rather than the rubric attached to it and never raises a type error. 520 to 532.
  • 2026-09-16: Three community additions merged, all from authors of the papers: the TaoLive harness-aware training report (@sunyuhan19981208, #15), Steer, Don't Solve and DRACO (@shubhamrgandhi, #16 and #17). The glosses now include the papers' own numbers; the TaoLive one leads with the finding that fixed-harness finetuning costs 7.7 points of instruction following while harness-state augmentation costs none. 517 to 520.
  • 2026-09-16: The contributors table is now collapsible like the sections, and sync_counts.py updates its count as well.
  • 2026-09-07: +20 papers posted since May 2026, two per section. LoopArena was suggested by one of its authors, @burgerseater, in #14. Several are negative results; in one, frontier agents take a planted shortcut in 57 per cent of runs even when the prompt asks them not to. 497 to 517.
  • 2026-09-03: Three papers added by @dukesun99, an author on all three (#13): Corpus2Skill to Memory, OrchMAS to Multi-Agent Systems, and Information Retrieval Misses the Mark for LLM Agents to Surveys & Position Papers. That last one is the list's first SSRN paper. The Corpus2Skill gloss now gives its results on eleven datasets, where corpus navigation loses on three. 494 to 497.
  • 2026-08-25: ForeDreamer added to Memory by @zhongzero, who is one of its authors (#12). The gloss now starts with what sets the system apart: it turns web evidence into structured memory before forecasting instead of passing retrieval results straight to the agent. 493 to 494.
  • 2026-08-20: +20 papers from August 2026, two per section. Several are negative results, such as a multilingual gap that doesn't close as models get larger. 473 to 493.
  • 2026-08-20: Two community additions merged: Dr. Bench (@reacher-z, #7) and CityReal (@Nicolas99-9, #10). 471 to 473.
  • 2026-08-12: Four community additions merged: Ouroboros (@razzant, #8), CORAL (@BobbyZhouZijian, #9), and WebAgent plus the world-model-augmented web agent (@JEONGSEJIN, #6). One code link was dropped that had been copied from the entry above it. 467 to 471.
  • 2026-08-08: +16 papers from August 2026 across all ten sections, with official repos linked where they exist. 451 to 467.
  • 2026-08-08: Nine glosses had been cut off mid-sentence by an earlier backfill. They're fixed, and Agon has the [code] link that @WhymustIhaveaname supplied in #5. There's a check for this in scripts/check_glosses.py now.
  • 2026-08-06: The survey is posted on Preprints.org with DOI 10.20944/preprints202608.0265.v1. The citation block, CITATION.cff and the header link now point at the versioned record instead of the PDF in this repository.
  • 2026-07-31: +30 papers from July 2026, three per section, with official repos linked where they exist. 421 to 451.
  • 2026-07-26: ClawBench added to Evaluation & Benchmarks by @reacher-z, who helps maintain it. 420 to 421.
  • 2026-07-25: First community contributions: +6 agent-memory systems (MemoryOS, Zep, Nemori, MemOS, G-Memory, ACE) from @jinmang2, and +3 research-agent and orchestration papers (AutoNumerics, OptimAI, PerspectiveGap) from @WhymustIhaveaname.
  • 2026-07-19: +78 papers from January to July 2026 across all ten sections, with official repos linked where they exist.
  • 2026-07-19: +30 papers from January to May 2026 (three per section), with official repos linked where they exist.
  • 2026-07-16: +50 papers from June and July 2026 across all ten sections, with official repos linked where they exist.
  • 2026-07-12: +42 papers from 2026 across all ten sections, a new 10 to Watch (2026) section with live star counts, and live star badges on the related lists. 211 to 253.
  • 2026-07-09: +27 papers on agentic RL, protocols, deep research, and frontier evaluation and safety. 184 to 211.
  • 2026-07-08: Initial release. 184 annotated papers, organized by the survey's taxonomy.