By The AI Bar  · 

What Is AI Hallucination in Law?

Why AI models fabricate case citations, why legal work is especially exposed, and what's happened to lawyers across the UK, US, Singapore and India who filed them unchecked.

AI hallucination in law is when an AI tool generates a case citation, quotation, statutory reference, or legal proposition that sounds authoritative and is formatted correctly, but is partly or entirely fabricated. It isn’t a bug in the ordinary sense; it’s a large language model doing exactly what it’s built to do — predicting the next plausible word — when it has no reliable answer to give. In legal practice, that produces cases that don’t exist, quotations no judge ever wrote, and holdings that misstate what a real case actually decided, all dressed in the confident, professional register that makes them hard to catch without checking.

Here’s why it happens, why legal work is especially exposed to it, and what’s happened to lawyers across the UK, US, Singapore, and India who filed it unchecked.

Why AI Models Hallucinate

A large language model doesn’t look anything up. It generates text by predicting, one token at a time, what’s statistically likely to come next based on patterns learned from its training data. Most of the time, for well-represented topics, this produces accurate-sounding and genuinely accurate text. But when the model reaches a gap — a case that doesn’t quite exist, a citation it never actually saw, a proposition nobody has written down — it doesn’t stop and say so. It fills the gap with something shaped like a real answer, because generating plausible-shaped text is the entire mechanism.

Some researchers prefer the term “confabulation” over “hallucination” for exactly this reason: the model isn’t malfunctioning or seeing things. It’s doing good-faith gap-filling, with no internal mechanism to flag the difference between a fact it retrieved and a fact it invented. The output looks identical either way.

Legal citation is a near-perfect setup for this failure mode. Case names follow predictable patterns (Party A v Party B). Citations follow rigid, learnable formats (volume, reporter, page, year). Judicial language has a recognisable register: measured, Latinate, procedurally formal. An AI model has seen enough real examples of all three that it can generate a fake one indistinguishable, on the page, from a real one. Unlike a hallucinated fact in casual conversation, a hallucinated citation doesn’t read as strange. It reads as exactly what a citation is supposed to look like.

This is why the failure is so dangerous in legal work specifically: the format is the plausibility. A study cited by Stanford researchers found that general-purpose chatbots hallucinate on a striking share of specific legal questions, with estimates in the range of 58% to 88% depending on the tool and question type. Legal-specific tools that ground answers in retrieved source material perform meaningfully better but do not eliminate the risk entirely.

How to Spot a Hallucinated Citation

There is no reliable way to spot a hallucination by reading it; that’s the entire problem. The only reliable method is verification against a primary source:

  1. Search the case name and citation directly in a verified database (Westlaw, LexisNexis, BAILII, SCC Online, or the relevant official court repository).
  2. Confirm the citation actually corresponds to the case name; AI tools sometimes combine a real case name with a citation that belongs to a different case entirely.
  3. Confirm any quoted language actually appears in the judgment, not just that the case exists.
  4. Confirm the holding described actually matches what the case decided, not a plausible-sounding paraphrase.

A citation that “looks right” on formatting alone tells you nothing. Every case below involved citations that looked right.

What’s Actually Happened to Lawyers Who Didn’t Check

Courts across multiple jurisdictions have now sanctioned lawyers for filing unverified AI-generated citations. The pattern is consistent: the tool produced the fabrication, but the lawyer who filed it bore the consequence.

England and Wales. In Ayinde v Haringey LBC and Al-Haroun v Qatar National Bank [2025] EWHC 1383 (Admin), decided together by the Divisional Court on 6 June 2025, the court (Dame Victoria Sharp P and Johnson J) held that “freely available generative artificial intelligence tools, trained on a large language model such as ChatGPT are not capable of conducting reliable legal research” and warned that lawyers relying on unverified AI output risk penalties up to contempt of court. In Ndaryiyumvire v Birmingham City University, a wasted costs order followed two fictitious AI-generated cases in a filing, with HHJ Charman treating it as a firm-level management failure. In 2026, Bournemouth Family Court ordered a lay advocate to be publicly named after filing a skeleton argument citing four cases that did not exist.

United States. A federal magistrate judge in Oregon fined two lawyers a combined $110,000 — the largest such penalty in the state’s history — after they filed briefs over a five-month period containing 15 fabricated case citations and 8 invented quotations; the judge called the case “a notorious outlier in both degree and volume.” In Mississippi, a federal judge removed four lawyers from a case in June 2026 after AI hallucinations appeared in their filings. In Nebraska, attorney Greg Lake filed a divorce appeal brief in which 57 of 63 citations were defective, including 20 entirely fabricated cases; the Nebraska Supreme Court suspended him in April 2026. Legal-industry trackers now log well over a thousand court decisions worldwide addressing AI-hallucinated material in filings.

Singapore. Justice S. Mohan of the Singapore High Court ordered two lawyers to personally pay S$5,000 each after discovering that written submissions cited two authorities that did not exist: one combining a real case name with a citation belonging to an entirely different case, the other fabricated outright. The judge rejected the lawyers’ initial offer of a S$1,500 costs contribution as inadequate given the seriousness of the issue.

India. In July 2026, in Pooja Ramesh Singh v Jammu and Kashmir Bank Ltd, the Supreme Court (Narasimha and Aradhe JJ) set aside NCLT and NCLAT orders that had relied on fabricated AI-generated precedents, holding that a decision founded on hallucinated precedents is no decision in the eyes of the law, and directed the Bar Council of India to constitute a committee and prescribe guiding principles, including the disciplinary consequences of breach. The Court declared zero tolerance for the citing of unverified AI-generated material by the Bar and the Bench alike. That matters because India, unlike the UK, currently has no formal bar guidance on AI use, meaning the judiciary is regulating the issue ahead of the profession’s own rules.

How to Prevent It

Use tools that ground answers in retrieval, not pure generation. Tools that search a live, verified source (legislation databases, case law repositories, your own document set) before generating an answer, rather than producing text purely from training data, substantially reduce hallucination rates. They don’t eliminate the risk, which is why verification remains necessary regardless of tool.

Build verification into the workflow, not into memory. A rule that everyone intends to follow but nobody is required to document gets skipped under deadline pressure. Firms with the fewest hallucination incidents tend to have a mandatory, recorded verification step, not just a policy that says “please check.”

Treat AI output as a first draft from a very well-read but occasionally dishonest junior. Confident phrasing is not evidence of accuracy. The more fluent and certain an AI-generated citation sounds, the more it deserves the same scrutiny as any other unverified claim in a first draft.

Supervise, don’t just permit. In several of the cases above, courts placed responsibility on supervising lawyers and firms, not only the individual who typed the prompt. A firm’s AI policy and training need to reach anyone who might touch a filing, not just the most senior person who approved using AI in the first place.

Frequently Asked Questions

What is an AI hallucination in simple terms? It’s when an AI tool generates false information (a case, quote, or fact) that sounds correct and is formatted correctly, but doesn’t exist or is inaccurate. It happens because the AI predicts plausible text rather than retrieving verified facts.

Can AI hallucinations really lead to a lawyer being sanctioned? Yes. Courts in England and Wales, the United States, Singapore, and India have all sanctioned lawyers or their firms for filing AI-generated content containing fabricated citations, ranging from costs orders to suspension from practice.

Do legal-specific AI tools still hallucinate? Yes, though less often than general-purpose chatbots. Tools that ground their answers in retrieval from live, verified legal databases substantially reduce hallucination rates compared to general-purpose models, but do not eliminate the risk. Verification remains the lawyer’s responsibility regardless of which tool is used.

How common is AI hallucination in legal research? Studies have found general-purpose AI chatbots hallucinate on a large share of specific legal questions; estimates range widely by tool and question type, with some studies citing rates as high as 58% to 88% for certain query categories.

Who is responsible if an AI tool hallucinates a citation in a court filing? The lawyer who filed it, not the AI tool or its provider. Every reported sanctions case has held the filing lawyer, and in several instances the supervising firm, responsible for verifying AI output before it reaches a court or a client.

For a structured verification workflow you can apply to every AI-assisted research task, see The AI Bar’s Litigation Support with AI.