An accuracy failure is an agent confidently giving wrong information, such as an invented policy, price or fact, that someone then relied on.
What are real examples of AI agents giving wrong answers that caused harm?
As of 21 October 2026, the Agent Failures Index records 12 accuracy incidents, 13% of the 90 it holds. 9 are case studies of an organisation hurt by its own agent; the other 3 involve agent products themselves. 9 were rated critical or high severity. The most recent: Migrating Copilot code review to shared CLI tools made review quality worse until instructions were rewritten (10 July 2026).
GitHub swapped Copilot code review's purpose-built code-exploration tools for the shared grep/glob/view tools used by the Copilot CLI harness, expecting a clean upgrade, but benchmarks showed reviews got more expensive and caught fewer real issues because the surrounding instructions weren't rewritten for the new tools.
Blast radius
Benchmarked regression in review cost and issue-catch rate during the migration period, caught before broad rollout via internal evaluation rather than by customers.
Root cause
The tool layer was swapped without rewriting the workflow instructions around it, so instructions tuned for the old, more verbose tools didn't match how the new Unix-style tools should be driven.
How it was caught
Internal benchmarking comparing old vs. new tool sets flagged higher cost and lower issue detection before the regression reached general availability.
What changed
GitHub rewrote the tool-use instructions for how a reviewer actually reads a pull request, cutting average review cost by roughly 20% while restoring review quality.
Take-awayWhen migrating an agent to a shared tool harness, the prompts and workflow around the tools are part of the interface — swap them together, not the tools alone.
Crown Prosecution Service (UK)·generative AI legal research tool (unnamed)
CPS lawyers used generative AI for legal research on an extradition appeal to Romania; two completely fictitious legal authorities that were 'likely generated through the use of generative AI' were included in the CPS's grounds of opposition and then repeated in further submissions by junior counsel before the High Court.
Blast radius
Two fabricated case citations were part of formal court filings in more than one submission before the High Court; the CPS reviewed almost 80 other cases handled by the same lawyer, finding no other issues, and there was no impact on the case outcome because the errors were caught before the hearing.
Root cause
The hallucinated citations were not verified before formal filing and were copied uncritically from one document into another across multiple stages of review.
How it was caught
Not identified during the court's permission stage; only surfaced when questions were raised shortly before the hearing.
What changed
The CPS apologised to the court, carried out an internal review, and said it had reinforced its verification processes.
Take-awayEvery AI-assisted legal citation needs to be checked against a primary legal database before filing, no matter how many prior reviewers have already seen it.
Anthropic (Claude Code, Claude Agent SDK, and Claude Cowork users)·Claude CodeFixed
Anthropic shipped three separate changes to Claude Code's harness — lowering default reasoning effort, a context-clearing bug that repeated every turn instead of once, and a verbosity-reducing system prompt — each of which degraded coding quality, and their overlap made the cause hard to isolate.
Blast radius
A month of user reports of 'Claude getting dumber' across Sonnet 4.6, Opus 4.6 and Opus 4.7 in Claude Code, the Agent SDK, and Cowork; the API itself was not affected. Anthropic reset usage limits for all subscribers as redress.
Root cause
Three overlapping harness/prompt changes, each individually intended to cut latency or verbosity, combined to look like broad model degradation, and internal evals didn't reproduce the issue at first.
How it was caught
User reports accumulated over about a month before Anthropic could distinguish the pattern from normal feedback variance and trace it to the three changes.
What changed
All three changes were reverted or corrected by April 20, 2026 (Claude Code v2.1.116), and usage limits were reset for all subscribers.
Take-awayTreat agent-harness and system-prompt tweaks with the same regression testing as model releases — aggregate evals can miss quality regressions that show up only in real usage.
US Immigration and Customs Enforcement (ICE)·AI resume-classification tool used in ICE's hiring pipeline
During a hiring blitz to reach 10,000 new recruits, ICE used an AI tool to sort applicants by experience level; the tool flagged anyone with the word 'officer' anywhere in their resume — including compliance officers and people who simply said they wanted to become ICE officers — as a prior law-enforcement officer, routing them into a shortened four-week training track instead of the full eight-week in-person course covering immigration law and firearm handling.
Blast radius
According to officials cited by NBC News, a majority of new applicants were misflagged before the error was caught, meaning an unknown but large number of undertrained recruits were sent directly into the field to carry out immigration arrests.
Root cause
A keyword-matching AI classifier was used to make a safety-critical training-track decision for armed personnel without adequate validation or human review of applicants' actual experience.
How it was caught
Identified internally by ICE officials only after a majority of applicants had already been misrouted, and made public by NBC News reporting in January 2026.
What changed
Not disclosed in available sources.
Take-awayDon't let a simple AI classifier make an armed-deployment training decision on keyword matches alone — verify claimed experience against records before shortening safety-critical training.
West Midlands Police·Microsoft Copilot (used for police intelligence assessment)
Officers used Microsoft Copilot to help compile threat intelligence ahead of a Europa League fixture, and the tool hallucinated a violent clash involving Maccabi Tel Aviv fans that never happened; that fabricated account was folded into the intelligence picture used to advise Birmingham's Safety Advisory Group to ban Maccabi Tel Aviv fans from the Aston Villa match.
Blast radius
Away fans were banned from an international fixture, triggering a national political crisis; the Chief Constable retired amid the fallout, the force was referred to the Independent Office for Police Conduct, and a Home Affairs Committee inquiry found the force 'overly reliant on inaccurate and unverified information for decision making that proved wholly inadequate to stand up to subsequent scrutiny'.
Root cause
Generative AI output was used to help assess a public-safety threat level without being checked against primary source material, and it was accepted uncritically because it reinforced a pre-held narrative.
How it was caught
Not caught until after the ban and match, when journalists and the Parliamentary Home Affairs Committee scrutinised the underlying 'evidence' and found the described violence had not occurred as claimed.
What changed
The force's acting Chief Constable paused and 'turned off' use of Copilot pending the IOPC investigation; the Home Affairs Committee recommended governance reforms, including ending the practice of councillors sitting on Safety Advisory Groups.
Take-awayNever let unverified generative-AI output feed into a high-stakes public-safety or civil-liberties decision without independent verification against primary sources.
Deloitte Australia / Department of Employment and Workplace Relations (DEWR)·an in-house generative AI tool chain (Azure OpenAI GPT-4o) used to draft the report
Deloitte used a generative AI large language model tool chain licensed by DEWR to help write an independent assurance review of the government's welfare compliance IT system, and the resulting report contained fabricated academic references, an invented quote attributed to a Federal Court judgment in the robo-debt case, and citations to papers that do not exist.
Blast radius
A university academic found over a dozen fabricated citations and quotes in the published report; Deloitte had to reissue corrected versions twice and partially refund the Australian government $440,000 (AUD) contract fee.
Root cause
AI-generated content in a client deliverable was not verified against real sources before publication and sign-off.
How it was caught
An external academic, Chris Rudge of the University of Sydney, spotted the fake citations after the report was already published, not through Deloitte's own review.
What changed
Deloitte reissued a corrected version of the report and settled the matter with the client via a partial refund; DEWR said the report's substantive recommendations were unchanged.
Take-awayNever publish an AI-assisted deliverable to a client without independently verifying every citation and quotation against the primary source.
Anthropic (Claude API, Bedrock, and Vertex AI users)·Claude API serving infrastructureFixed
A misrouting bug sent short-context Sonnet 4 requests to servers configured for the 1M-token context window, a TPU serving misconfiguration caused token-generation corruption (e.g. stray Thai/Chinese characters or syntax errors in English/code responses), and a separate XLA:TPU compiler bug hit token selection for Haiku 3.5 and other models.
Blast radius
At the worst hour on August 31, 2025, 16% of Sonnet 4 requests were misrouted; about 30% of Claude Code users hit at least one degraded response during the period. Output corruption affected Opus 4.1, Opus 4, and Sonnet 4 requests between August 25 and September 2.
Root cause
Three separate infrastructure changes (routing logic, a TPU serving misconfiguration, and a compiler-triggering optimization) overlapped in time, and sticky routing meant affected users kept hitting the same bad server, producing confusing, inconsistent reports.
How it was caught
User reports of degraded responses beginning in early August were initially indistinguishable from normal feedback variance; a formal investigation opened in late August traced them to three distinct bugs.
What changed
Routing logic was corrected and rolled out across platforms by September 18; the TPU misconfiguration was rolled back September 2 with new output-anomaly detection tests added; the compiler bug was mitigated for affected models.
Take-awayWhen you serve one model across multiple hardware backends, add automated output-quality canaries per backend, since sticky routing can hide a bug from most users while a minority get every response from the broken path.
Morgan & Morgan, P.A.·MX2.law, the firm's proprietary in-house AI legal research platform
An associate at Morgan & Morgan used the firm's own AI platform, MX2.law, to draft pretrial motions in a Wyoming product-liability suit against Walmart; the tool invented eight of the nine cases cited, and the motions were e-signed and filed without anyone checking the citations.
Blast radius
Three Morgan & Morgan attorneys were sanctioned by a federal court and fined a combined $5,000; the motions had to be withdrawn, and the episode was later cited against the same attorney in a separate Massachusetts court application.
Root cause
No human review step existed between the AI tool's output and the attorneys' e-signature on a court filing.
How it was caught
Opposing counsel for Walmart flagged the bogus citations to the court, prompting the judge to demand an explanation.
What changed
Not disclosed — a later court noted the firm had not described any new review process or change to how MX2.law is used since the sanctions.
Take-awayAn in-house AI tool used in core operations still needs a mandatory human verification gate before its output leaves the building.
UnitedHealthcare / UnitedHealth Group (via subsidiary naviHealth)·nH Predict AI modelUnresolved
UnitedHealthcare used its nH Predict AI model, built on a database of six million prior patients, to predict how much post-acute care a Medicare Advantage patient "should" need and to pinpoint when to cut off payment, allegedly overriding treating physicians' recommendations.
Blast radius
A federal class action alleges named plaintiffs faced tens of thousands of dollars in out-of-pocket care costs and worsened health after coverage was cut early; the suit says patients who appealed denials won more than 90% of the time, and claims span patients in over a dozen states.
Root cause
An automated predictive model was used to drive high-stakes coverage cutoff decisions with, plaintiffs allege, insufficient individualized clinical review.
How it was caught
A STAT News investigation and a subsequent wrongful-denial class-action lawsuit against UnitedHealth Group.
What changed
A federal judge dismissed five of seven counts in February 2025 but allowed breach-of-contract claims to proceed; litigation is ongoing and UnitedHealth disputes the claims, saying the tool is only a planning guide, not a coverage-determination system.
Take-awayIf a predictive model informs a decision that can end someone's care or benefits, enforce meaningful human clinical review before the model's output becomes the action.
French government / Linagora-led consortium·Lucie, a French-language AI chatbot backed by a French government AI initiative
Shortly after Lucie launched, users found it gave nonsensical answers to basic questions, including miscalculating simple arithmetic and, when asked about 'cow's eggs,' replying that they are edible eggs produced by cows.
Blast radius
The chatbot was taken offline days after its public launch amid widespread online ridicule; the developer consortium acknowledged it had been released prematurely.
Root cause
The model was released publicly while still, in the developer's own words, an early-stage research project rather than a validated production system
How it was caught
Users shared erroneous answers on social media within days of launch, driving public ridicule that forced a pause
What changed
Linagora said it would update the model, test a private beta, and relaunch only after further validation
Take-awayDon't launch a public-facing conversational agent from a research-stage model without a private validation phase first
DoNotPay, Inc.·DoNotPay AI 'robot lawyer' chatbot and document-generation tool
DoNotPay marketed its AI chatbot as able to generate legally valid documents and give advice that could substitute for a human lawyer, including suing for assault without a lawyer, but per the FTC complaint the company never tested whether the AI's output matched the quality of an actual attorney or had lawyers review its legal features.
Blast radius
Consumers who subscribed between 2021 and 2023 relied on unverified AI-generated legal documents and advice for real legal matters; the FTC's final order required $193,000 in monetary relief to consumers and mandatory notice to affected subscribers.
Root cause
The company sold and advertised an AI agent as attorney-equivalent for consumer legal problems without substantiating the claim or having qualified attorneys test the agent's accuracy.
How it was caught
FTC investigation and enforcement action as part of its 'Operation AI Comply' sweep against deceptive AI marketing claims.
What changed
Final FTC order, approved 5-0 on January 16, 2025, bars DoNotPay from claiming its service performs like a human lawyer without competent evidence, and requires consumer redress and notice to past subscribers.
Take-awayDon't let a customer-facing agent make professional-service claims (legal, medical, financial) that haven't been validated against real expert performance — regulators will treat those claims as testable and will hold the deploying company liable.
UK Department for Work and Pensions·DWP automated risk-scoring algorithm for housing benefit claims
DWP's automated risk-scoring algorithm flagged over 200,000 housing benefit claims as high risk for fraud or error and triggered investigations, even though roughly two-thirds of the flagged claims turned out to be legitimate.
Blast radius
Hundreds of thousands of claimants were put through unnecessary fraud investigations and stress, and public funds and investigator time were wasted chasing false positives, despite the algorithm having shown initial success in a pilot before real-world deployment.
Root cause
The model's real-world performance fell far short of its pilot results once scaled to the full claimant population, reflecting overreliance on an automated system in welfare administration.
How it was caught
Surfaced through investigative reporting on DWP's fraud-detection performance, not through internal DWP disclosure.
What changed
Not disclosed in available sources; DWP's wider use of AI in benefits decisions has remained under continued scrutiny, including a later DWP 'fairness analysis' that found bias in a related fraud-detection system.
Take-awayA fraud-detection model's pilot accuracy does not predict its real-world false-positive rate at national scale — validate continuously after rollout, not just before it.
From the Agent Failures Index: every entry is traced to a primary source, and each incident is filed under one failure type. See also the monthly reports.