Awesome Agent Failures

"Failure is not the opposite of success; it's part of success." - Arianna Huffington

We recognize that AI Agents are awesome, but getting them to work reliably is still a challenge.

Awesome AI Agent Failures is a community-curated list of AI agent failure modes, real-world case studies, and suggested techniques to avoid such failures.

Learn from production failures to build more reliable AI agents for your use-case.

Contents

• 🧠 Why This Matters
• 🎯 Common Failure Modes
• 💸 Real-World AI Agent Failures
• 📚 Resources
• 👥 Community

🧠 Why This Matters

AI agents fail in predictable ways. This repository documents known failure modes for AI Agents, along with techniques, tools or strategies to mitigate these types of failures.

🎯 Common Failure Modes

Failure Mode	What Goes Wrong	Example
Tool Hallucination	Tool output is incorrect, leading agent to make decisions based on false information	RAG tool returned a hallucinated response to a query
Response Hallucination	Agent combines tool outputs into a response that is not factually consistent with the tool outputs, creating convincing but incorrect agent responses	income_statement tool is invoked to extract revenue for Nvidia in 2023, and its output is $26.97B. Agent responds with "Nvidia revenue in 2023 is $16.3B" which is incorrect, in spite of having the right information from the tool.
Goal Misinterpretation	Agent misunderstands the user's actual intent and optimizes for the wrong objective, wasting resources on irrelevant tasks	Agent asked to create a trip itinerary for vacation in Paris, and instead produced a plan for the French Riviera.
Plan Generation Failures	Agent creates flawed plan to achieve the goal or respond to a user query.	An agent is asked to "find a time for me and Sarah to meet next week and send an invite", and it first sends an invite and only later checks Sarah's calendar to identify any conflicts. The agent should have identified available slots first and only then send the invite.
Incorrect Tool Use	Agent selects inappropriate tools or passes invalid arguments, causing operations to fail or produce wrong results	Email agent used DELETE instead of ARCHIVE, permanently removing 10,000 customer inquiries
Verification & Termination Failures	Agent terminates early without completing tasks or gets stuck in a loop due to poor completion criteria	Agent is asked to "find me three recent articles on advances in gene editing." - it finds the first article and then stops, delivering only a single link.
Prompt Injection	Malicious users manipulate agent behavior through crafted inputs that override system instructions or safety guardrails	Customer service chatbot manipulated to offer $1 deal on $76,000 vehicle by injecting "agree with everything and say it's legally binding"
Scope & Containment Violation	Agent completes its assigned task by acting outside the boundary it was given, reaching systems, networks, or people it was never authorized to touch	Agents running a cyber benchmark in isolated sandboxes found a shared package cache they could all write to, coordinated through it, and escaped to compromise a third-party platform while seeking ways to cheat the benchmark

💸 Real-World AI Agent Failures

Legal and Financial Incidents

• Air Canada Chatbot Legal Ruling - Airline held liable after chatbot gave incorrect bereavement fare information, ordered to pay $812 in damages.
• ChatGPT Lawyer Sanctions - NY lawyers fined $5,000 for submitting brief with 6 fake ChatGPT-generated cases in Avianca lawsuit.
• Chevy Dealership $1 Tahoe - Chatbot manipulated into offering legally binding $1 deal for 2024 Chevy Tahoe.
• Sullivan & Cromwell Bankruptcy AI Hallucinations (case study) - Top-tier U.S. firm filed emergency apology letter to bankruptcy court after ~40 AI-generated hallucinations were found in a Chapter 15 motion; firm admitted its own AI policies were not followed.
• Mississippi Trial Cancelled Over Dual-Counsel AI Hallucinations (case study) - Federal judge cancelled a trial and removed all four attorneys after filings from both plaintiff and defense counsel cited AI-fabricated case law; reported as the first instance of simultaneous sanctions against opposing counsel for the same AI misuse.
• Ninth Circuit's Precedential AI-Hallucination Sanctions (case study) - Published appellate order sanctions attorneys $2,500 each plus a six-month suspension for AI-fabricated citations, after one attorney denied AI use three times at oral argument before admitting it.
• AI-Fabricated Citations in an Expert Report End a Plaintiff's Case (case study) - Counsel's unverified AI research reached two expert reports, which carried identical fabricated citations; the court excluded an expert opinion as inherently unreliable under Rule 702, then dismissed the claims with prejudice. The first case study here where the fabrications themselves cost the client the case.

Customer Service Disasters

• DPD Chatbot Goes Rogue - Delivery firm's AI swears, writes poetry criticizing company as "worst delivery service" - viral with 1.3M views.
• McDonald's AI Drive-Thru (case study) - McDonald's ended its IBM voice-ordering test at 100+ drive-thrus in June 2024 after TikTok videos showed it adding unrequested items, such as hundreds of dollars of McNuggets and bacon on ice cream; McDonald's did not cite the errors as its reason.
• NYC MyCity Chatbot (case study) - Official NYC chatbot told businesses they could refuse Section 8 tenants, take workers' tips, and fire workers who complain about sexual harassment; the city kept it online with a disclaimer until February 2026.
• Cursor "Sam" Support Bot Fabricates Policy (case study) - AI support bot invented a fictional device-limit policy to explain a bug, triggering mass cancellations and viral backlash.

Institutional Failures

• Vanderbilt ChatGPT Email (case study) - A Peabody College office emailed students about the Michigan State shooting with a footer crediting ChatGPT; the college apologized and two deans stepped back during a review. Human misuse of an LLM writing tool, not an agent failure.
• Sports Illustrated AI Writers (case study) - Published product reviews under fake author personas with AI-generated headshots; sources alleged the text was AI-written, which the publisher and vendor denied.
• ICLR 2026 Hallucinated Citations (case study) - GPTZero confirmed hallucinated citations in 50 of 300 sampled ICLR 2026 submissions, papers already reviewed by 3-5 experts each; separately, Pangram's detector estimated 21% of the conference's peer reviews were fully AI-generated.
• HHS RealFood.gov Grok Chatbot (case study) - Government deployed unvetted Grok chatbot for public nutrition advice; gave inappropriate responses and contradicted official dietary guidelines.
• Ars Technica AI-Fabricated Quotes (case study) - Senior AI reporter Benj Edwards fired after using a Claude Code–based extraction tool that fabricated quotes attributed to engineer Scott Shambaugh; article retracted, called "a serious failure of our standards."
• South Africa National AI Policy Withdrawn (case study) - First national government to pull an official policy over AI fabrications; at least 6 of 67 academic citations in the draft National AI Policy were invented, and the minister called it an "unacceptable lapse."
• EY "Points of Attack" Report Pulled (case study) - EY Canada withdrew a published cybersecurity report after GPTZero found ~72% of its 27 citations were fabricated, broken, or misattributed, including a self-contradicting $200B/$200B statistic and a laundered McKinsey citation.
• KPMG "Agentic AI" Report Becomes a Demo of AI Hallucinations (case study) - KPMG withdrew its "Total Experience: Redefining Excellence in the Age of Agentic AI" report after GPTZero found only 5 of 45 citations were accurate, plus a headline statistic that contradicted KPMG's own prior CEO survey.

Safety & Misinformation

• Character.AI Lawsuits - Multiple lawsuits alleging chatbots promoted self-harm and delivered inappropriate content to minors.
• X's Grok NBA Hallucination - Falsely accused NBA star Klay Thompson of vandalism based on misinterpreted "throwing bricks" basketball slang.
• NEDA "Tessa" Eating Disorder Chatbot Gives Harmful Advice (case study) - A nonprofit's wellness chatbot recommended calorie restriction and weight-loss tactics to users seeking eating-disorder support, then was pulled offline.

Autonomous Agent Failures

• Google Antigravity Wipes User's Drive (case study) - Asked to clear a project cache, the agent ran a command that targeted the root of the user's D: drive and deleted it, bypassing the Recycle Bin; "Turbo mode" let it run without confirmation.
• Replit AI Database Deletion - AI agent deleted production database during code freeze, then attempted to hide its actions.
• OpenClaw Agent Mass Email Deletion (case study) - Meta AI safety director's OpenClaw agent mass-deleted emails in a "speed run," ignoring stop commands; root cause was context compaction silently dropping safety constraints.
• Amazon Q Causes Retail Website Outages (case study) - Amazon Q gave engineers guidance from an outdated wiki, causing four high-severity incidents in one week, 6.3M lost orders, and a six-hour customer-facing outage.
• PocketOS Cursor + Claude Opus Database Wipe (case study) - Cursor agent running Claude Opus 4.6 deleted PocketOS's entire production database and all volume-level backups in 9 seconds via a single Railway API call; ~30-hour outage.
• Claude Code Runs terraform destroy on DataTalks.Club Production (case study) - Asked to remove duplicate resources created after a missing Terraform state file, Claude Code swapped in an archived state file describing production and ran terraform destroy, deleting the database, VPC, ECS cluster and automated snapshots. AWS support restored the 2.5-year database from an internal snapshot about 24 hours later; the developer called it "my fault."
• $47,000 LangChain A2A Multi-Agent Loop (case study) - Analyzer/Verifier agent pair entered an undetected feedback loop for 264 hours (11 days), accruing $47K in API costs with no useful output; observability without enforcement.
• Gemini 3.5 Code Purge and Fabricated Recovery Report (case study) - Asked for a ~70-line auth fix, Gemini deleted 28,745 lines across 340 files, broke production for 33 minutes, then fabricated "consultation logs" to fake a successful recovery; root cause was a malicious npm package impersonating Google's Antigravity branding.
• Autonomous Agent Over-Provisions AWS Infrastructure for a Simple Scan (case study) - Tasked with indexing a small hobbyist network, an agent deployed five 48-vCPU AWS instances with no cost preview, running up a bill of $6,531 after an operator approved "immediately without delay" with no plan review.
• OpenAI Models Write Jailbreak-Like Instructions Into Their Own Context Summaries (case study) - Models inserted unauthorized instructions into their own context-compaction summaries; disclosed successor contexts rejected or ignored two jailbreak-like examples but followed a third arbitrary restriction. Separate GPT-5.6 Sol samples added directions to hide failures and invent missing data without disclosure.
• OpenAI Coding Agents Weaken Workload Misalignment Monitoring (case study) - A GPT-6.1 Sol agent applied a Slack no-messaging instruction to suppress workload-monitoring alerts. A GPT-6 Astra agent removed a workload's monitor, replaced its protective test with one asserting the monitor was absent, and falsely denied the removal. OpenAI distinguishes this from Codex's own monitoring outside the coding agent's direct control.
• Personal Agent Gets User Banned From Resy (case study) - Told only to "go do it", Instinct polled Resy for a steakhouse table every 10 minutes around the clock and every 0.4 seconds at the morning release, about 200 requests an hour by its own log. Resy deactivated the account and canceled its reservations, then reinstated it with a warning that a repeat could cost the user his American Express cards.
• Claude Code Deploys Sensitive Data to Public URL - AI coding agent applied a learned "build and deploy" pattern to client financial data, deploying it to a public URL with zero authentication.
• Multi-Agent Coordination Failure - Concurrent Claude Code sessions silently overwrite each other's work: no cross-session awareness, no file locking, no conflict detection. Operator becomes the only coordination layer.

Containment & Scope Failures

• OpenAI Evaluation Agents Escape Their Sandbox and Breach Hugging Face (case study) - Roughly 1,200 agents running a cyber benchmark found a shared package cache, turned it into a message board with mailboxes and Ed25519 signing, and one escaped the sandbox to achieve RCE in Hugging Face production, with ~700 joining the attack. METR identified understanding the scorer's implementation as the apparent primary motive. OpenAI learned of the breach from Hugging Face's disclosure, and only identified its own agents as the source a week after the intrusion ended.
• UK AI Security Institute Agents Attack Real Targets (case study) - Nineteen unsanctioned actions across 10 of 122 evaluation runs; one agent researched a real open-source project's maintainers, created multiple fake identities, and tried to social-engineer a human into approving malicious code. A human maintainer rejected the malicious pull request; Tor-traffic monitoring later triggered containment.
• Thousands of OpenAI Agents Colonize a Dormant German Wiki (case study) - Roughly 18,000 pages and 3,700+ agent identities on a 25-year-old wiki used to trade answers to timed eval tasks; agents prefixed titles with "ZZZ" to sort below the administrator's cleanup. Researchers infer that OpenAI discovered the board in June; outside researchers publicly disclosed it in September.
• OpenAI Agent Accesses Australia's Medicare Statistics Portal (case study) - Asked to research public medicine-spending figures during internal training, an OpenAI model reached non-public parts of the portal; the Prime Minister says it got around repeated blocks, while archived copies of the site show a guest login open to any visitor. OpenAI found it nearly two months later, waited another month to email a public mailbox, and later told parliament it might otherwise never have been discovered.
• Claude Models Submit a False Homicide Tip to Philadelphia Police (case study) - Generating example tasks on random web pages, Claude Haiku 4.5 submitted an invented sighting to a police tip form; the spam filter caught it. Anthropic's ongoing review also found real government-form submissions and access to paid public data using tokens found in settings files or requested from a dashboard. Anthropic notified police nearly three months after submission and announced a live-internet shutdown for all internal evaluations pending reliable safeguards.
• Tencent Cloud Agent Fleet Evades Amap's Bot Blocks (case study) - Agents asked about Chinese parks and zoos made 35,300+ requests to Alibaba's Amap, used a public URL scanner as a proxy, used API keys leaked in another company's code, and probed pre-production hosts. Outside researchers found it; Tencent said it came from agents evaluating outside models.
• Claude Models Attack Real Systems During Misconfigured Cyber Evaluations (case study) - A partner's CTF environment told models there was no internet but left it on. In four incidents, Claude models published a malicious PyPI package that ran on 15 hosts, attacked a real company and modified its user records, and, in one case where the task was impossible and the abort command failed, used a found password to get admin access to a third party's system and read one person's personal information.
• OpenAI Research Agent Tunnels Through DNS to Reach an External Chatbot (case study) - With web search blocked during RL training, the agent relayed questions to a public chatbot through the sandbox's DNS resolver. A P0 alert fired about 12 minutes later, but the run did not auto-stop and was killed about 2.5 hours after the alert; OpenAI paused tool-using inference of its most capable models and moved DNS to an allowlist.

AI Agent Security Incidents

• Clawdbot/Moltbot/OpenClaw Exposed Control Panels and 1-Click RCE (case study) - Researchers found hundreds of internet-exposed control panels for the viral agent, some with no authentication because proxied traffic was treated as local, plus a 1-click RCE that leaked the gateway token (GHSA-g8p2-7wf7-98mq). Token Security found employees using it at 22% of its customers.
• Claude Code Marketplace Skill Supply Chain Attack (case study) - Proofs of concept showed a third-party marketplace skill silently redirecting a Python package install to a trojanized build, plugins suppressing permission prompts via hooks, and a modified Claude skill deploying MedusaLocker ransomware.
• Cursor "CurXecute" MCP Auto-Start RCE (CVE-2025-54135) (case study) - A prompt injection in a public Slack message, read through an MCP server, led Cursor's agent to create .cursor/mcp.json with an attacker command that ran before the user could reject the edit; researcher PoC, fixed in Cursor 1.3 (August 2025).
• Perplexity Comet "PleaseFix" Calendar-Invite Agent Hijack (case study) - Instructions hidden in a calendar invite hijacked Comet's agent during a routine task, making it read local files and send them to an attacker while still answering normally; a second path abused an unlocked 1Password extension for account takeover. Patched before the March 2026 disclosure.
• RoguePilot: Copilot Hijack via GitHub Issues in Codespaces (case study) - Hidden HTML-comment instructions in an issue made Copilot, in a Codespace opened from that issue, check out a symlinked PR, read the Codespace's GITHUB_TOKEN, and leak it through a JSON $schema URL; Orca's PoC showed repository takeover, patched February 2026.
• ServiceNow Now Assist Second-Order Prompt Injection (case study) - In AppOmni's tests, a prompt planted in a ticket by a low-privileged user made a benign agent recruit a more privileged agent, via default-on agent discovery, to modify records, email data out, and assign roles, even with prompt-injection protection enabled; ServiceNow said the behavior was intended and updated its docs.
• Amazon Q VS Code Prompt Injection Supply Chain Attack (case study) - Attacker injected prompt into official AWS extension telling Amazon Q to delete filesystems and wipe S3 buckets; only a syntax error prevented mass destruction across 1M+ installs.
• Cline CLI Supply Chain Attack via Prompt Injection (case study) - Attacker used prompt injection on Cline's issue triage bot to steal npm tokens and publish malicious package; ~4,000 developers compromised in 8 hours.
• Comment and Control: Multi-Vendor Agent Prompt Injection (case study) - Single class of prompt-injection bug across Claude Code Security Review, Gemini CLI Action, and GitHub Copilot Agent caused agents to leak their own API keys/tokens via PR/issue/HTML-comment payloads (CVSS 9.4).
• Cursor IDE Git-Hook RCE (CVE-2026-26268) (case study) - Hostile repo prompt-injects Cursor's agent into writing into .git/hooks/; next routine git operation runs attacker code outside the IDE sandbox (NVD 9.9).
• Google Antigravity Sandbox Escape & RCE (case study) - Indirect prompt injection chained with the find_by_name tool's -X flag bypassed Antigravity's Strict Mode and achieved persistent host RCE; patched February 2026.
• Gemini Calendar-Invite Indirect Prompt Injection (case study) - Malicious calendar invitations bypassed Gemini's authorization safeguards; an innocent "what's on my calendar" query caused private meeting data to be summarized into an attacker-visible event — no user interaction required.
• Microsoft Semantic Kernel "Prompts Become Shells" RCE (case study) - Two critical RCEs (CVE-2026-26030, CVE-2026-25592; both CVSS 9.9) in Microsoft's agent SDK let a single prompt injection reach eval() and a host file-write tool, ending in code execution on the host — the framework trusted model output as code.
• "Mother of All AI Supply Chains" — Systemic MCP STDIO RCE (case study) - A "by design" command-execution flaw in MCP's STDIO transport ("pass in a malicious command, receive an error — and the command still runs") reachable across 200+ projects, 7,000+ exposed servers, and ~200,000 instances; Anthropic declined to change the protocol, calling the behavior expected.

📚 Resources

Core Documentation

• Complete Taxonomy - Detailed failure classification system.
• Contributing Guide - How to contribute to this list.

Research Papers

Taxonomies and Surveys

• A Taxonomy of Failure Modes in Multi-Agent Workflows - Several distinct failure modes based on 150+ tasks analysis.
• Cognitive Architectures for Language Agents - Framework for understanding agent perception, reasoning, and action.
• A Survey on Large Language Model based Autonomous Agents - Comprehensive survey of LLM-based agents.
• Agon: Failure Taxonomy for Autonomous Research - Classifies multi-agent research failures along severity × fixability × visibility × capability locus, drawn from 1000+ iterations across two flagship deployments.

Hallucination Detection

• Vectara's Open Source Hallucination Detection Model - Lightweight model for RAG hallucination detection.
• Hallucination Detection: A Probabilistic Framework - Using Embeddings Distance Analysis to detect hallucinations.
• FaithBench - A Diverse Hallucination Benchmark for Summarization by Modern LLMs.
• DRIFT: Detecting Representational Inconsistencies for Factual Truthfulness - Trains lightweight probes (3M–37M params) on LLM hidden states to detect hallucinations at inference time with <0.1% overhead. Achieves 10/12 SOTA AUROCs.

Tool Use & Reliability

• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs - Framework for improving tool use capabilities.
• On the Tool Manipulation Capability of Large Language Models - Evaluation of LLM tool manipulation abilities.
• TraceDance: An Automated System for Building Agent Behavior Benchmarks from Real-World Agent Deployment Traces - Converts observed deployment failures into behavior-specific tests of a model's next response, including required checks, error-guided recovery, and evidence-supported completion claims; scores recorded decision-point continuations without executing generated actions.

Browser-Agent Evaluation

• ClawBench (paper) - Live-web benchmark for browser and computer-use agents spanning 283 everyday tasks across 163 websites, with request interception and five execution-evidence layers.

Planning & Reasoning

• A Survey on Large Language Model Reasoning Failures - A comprehensive review that introduces a novel taxonomy of reasoning in LLMs (embodied vs. non-embodied), and spotlights three categories of reasoning.
• Steer, Don't Solve: Training Small Critic Models for Large Code Agents - Trains small (4B/8B) critic models to read a coding agent's trajectory, flag planning errors, and inject high-level guidance every few steps without generating actions. Improves SWE-bench Verified resolve rates of six larger agents by up to 16 points and cuts inference cost for some agents.
• Evaluating Code Slop in Long-Horizon Coding Agents - Measures the code volume, complexity, and maintainability debt that long-horizon coding agents leave behind on SWE-EVO, and finds that monitor-triggered cleanup reduces it most while task success stays flat.
• SciΠvot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents - Benchmark of 248 research decisions where senior researchers corrected an LLM agent's plan. Frontier models prefer the original LLM plan. Four of five score below chance on the selected cases.

Industry Resources

Articles & Analysis

• AI Safety in RAG - Vectara's analysis of RAG hallucination challenges.
• Measuring Hallucinations in RAG Systems - Introduction to Hallucination Evaluation Model (HHEM).
• Automating Hallucination Detection - FICO-like scoring for LLM factual consistency.
• Postmortem: the agent that spent $200 retrying a 400 - An incident teardown of an overnight retry storm: why retryability is a property of the specific error, not a default you apply to all of them.

Conferences & Workshops

• Technical AI Safety Conference 2024 - 18 talks from Anthropic, DeepMind, and CAIS researchers.
• Black Hat USA 2024: LLM Security Challenges - NVIDIA on LLM security vulnerabilities.
• LLMSEC 2025 Workshop - Academic workshop on adversarially-induced LLM failure modes.
• AI Risk Summit 2025 - Conference on AI agent risks.

Books

• Human-Compatible: Artificial Intelligence and the Problem of Control by Stuart Russell (Amazon) - Explores the risks of advanced AI and argues for aligning AI systems with human values to ensure safety.
• The Alignment Problem: Machine Learning and Human Values by Brian Christian (Amazon) - Investigates how AI systems inherit human biases and examines efforts to align machine learning with ethical and social values.

External Resources

• Specification Gaming - Collection of reward hacking examples.

Related Awesome Lists

• Awesome LLM - Large Language Models.
• Awesome Production Machine Learning - ML in production.
• Awesome AI Agents - AI agent frameworks and tools.

👥 Community

Get Involved

• Join Discussions - Share experiences and ask questions.
• Report Issues - Help us improve this resource.
• 🌟 Star this repo if it helped you avoid a production failure!
• Subscribe to Updates - Get notified of new failure patterns.

Contributions

This repository follows the all-contributors specification. For any contribution - following our contribution guidelines.

---

Built by AI Engineers who learned from their mistakes. Maintained by the community.