Beyond the Mirage: Beware of Generative AI and Hallucinations
6.26.2026
The work of attorneys can be arduous. With demanding caseloads comes an openness to tools that can help us do our jobs more efficiently. The advent of online legal research was a game changer for attorneys decades ago. In recent years, generative artificial intelligence – AI that can create original content such as text, images, video, audio or software code in response to a user’s prompt or request – has begun to revolutionize legal practice.[1] This development has included the integration of AI into legal research and writing.
The focus here is on the risks inherent in popular generative AI models used to complete such tasks: They are prone to producing false legal information, so-called hallucinations, including false case citations and false reasoning, quotes and holdings. A nationwide epidemic of cases involving such fabrications has made the risks of unverified AI use well known. Hundreds of decisions have touched on this issue.[2] Thus, the legal profession has been alerted that blind faith in generative AI results is misplaced.
This article makes no attempt to be exhaustive as to the rapidly unfolding case law but does seek to highlight some emblematic decisions issued by state, federal, trial and appellate courts throughout the country over the last two years and to bring attention to the dangers of failing to check AI results.
Hallucinations and Sanctions
First New York Appellate Decision Imposing Sanctions for Hallucinations
Oral argument in a recent appeal dramatized how generative AI-based research can go wrong. A judge asked counsel, “What about the elephant in the room?” By that, the judge meant allegations that AI-generated fake cites appeared in counsel’s papers. Counsel demurred – the cites were not important to salient issues. The court pushed back, “They are important to us.” The landmark decision in question is Deutsche Bank National Trust Co. v. LeTennier.[3] In the emerging discussions on generative AI misuse, this Appellate Division, Third Department decision stands out as reportedly the first appellate-level case in New York State court addressing sanctions for generative AI misuse.[4]
LeTennier discusses the value of generative AI, noting that it promotes access to justice, saves costs and assists in the administration of justice. However, when results are not confirmed, opposing counsel and the court may be called upon to spend countless hours separating fact from fiction. Due to erroneous assumptions made by the relevant model, biases in the data used to train the model and other factors, generative AI may produce incorrect information, the LeTennier court observed.
So-called hallucinations include fake citations to nonexistent cases and citations to real cases but misrepresentations of the holdings. The generative AI fictions often favor the user who supplied the query that generated the case information – perhaps because generative AI models are more likely to give an incorrect response than to admit that they do not know something.[5] While fabricated citations are more likely to arise from generative AI than the human mind, attorneys’ distortions of the meaning of case law is nothing new.[6]
LeTennier involved the execution by the defendant of a note to borrow money. The note was secured by a mortgage on real property, the defendant defaulted on his obligations, the plaintiff initiated a mortgage foreclosure action and its motion for summary judgment was granted. The defendant then filed several motions, including one seeking vacatur under CPLR 5015 based on new evidence and fraud. The instant appeal challenged the denial of the motions. The reviewing court affirmed all challenged orders.
Counsel’s opening brief cited six cases that do not exist – a problem detected by counsel for the plaintiff. In opposing such relief, the defendant cited more fake cases and inapposite decisions and claimed that the nonexistent cases constituted citation or formatting errors.
While the reply brief acknowledged “a serious error,” it also included more fictitious cases. The appellate papers included 23 fake citations and many misrepresentations. Such conduct implicated rules against frivolous conduct and providing knowingly false statements to a tribunal. In determining an appropriate sanction, the LeTennier court was not impressed with counsel’s statement at oral argument that 90% of the citations were accurate. Many of the fake cases were cited after counsel had been alerted to the problem; remedial action and remorse were instead the proper path.
In the end, what is at issue in LeTennier is not so much emerging technology as the old-fashioned notion that attorneys must validate every fact and every cite before filing their papers. Artificial intelligence demands oversight via human intelligence. In LeTennier, counsel’s lapses resulted in a sanction of $5,000 for the hallucination-related misconduct. For pursuing a frivolous appeal, an additional $2,500 each was imposed against counsel and the defendant, who had filed several pro se motions.[7]
Southern District Court Enters Default Judgment as Punishment
A New York Southern District case tells a similar tale of an attorney not appreciating the gravity of the situation and obfuscating and repeating errors, rather than showing contrition and seeking absolution. The case, Flycatcher Corp. Ltd. v. Affable Ave. LLC,[8] involves federal claims of trademark infringement and unfair competition.
The defendant’s brief supporting a motion to dismiss contained false citations. As in LeTennier, in this case, opposing counsel discovered the hallucinations and alerted the court, stating that perhaps the defendant’s counsel had used ChatGPT or another AI model, resulting in 13 fake citations and eight real cases with fictitious quotes.
As in LeTennier, counsel in Flycatcher continued to use flawed research methods and failed to check cites. Counsel again presented false citations both in response to the court’s order to show cause regarding sanctions and in a new brief.
The offending attorney in Flycatcher offered a song and dance about how these mistakes occurred and minimized the errors. Counsel’s unpersuasive defense: “Only” 14 out of 60 cases were hallucinations. The punishment for the “insouciant approach” and bad faith: the striking of the problem submissions and the entry of a default judgment against the defendant. Further, counsel for the plaintiffs was directed to submit an application for attorney’s fees to be paid by the defendant’s counsel.
Law Firms’ Duties: New York Trial Court
Among enlightening New York decisions considering potential discipline of attorneys for errant citations is Cassata v. Michael Macrina Architect, P.C.[9] In several ways, that case is striking. It details troubling ethical lapses but concludes that no disciplinary investigation is warranted;[10] probes ethical rules implicated; and explains the duties of supervising attorneys and law firms.
The Cassata litigation against an architectural firm, sounding in professional malpractice and breach of contract, sprang from an alleged house collapse. When the plaintiff sought to strike affirmative defenses, the defendant opposed by plagiarizing a brief generated by AI, according to the plaintiff.
At an ensuing sanctions hearing, the attorney author and her supervising attorney apologized. The junior attorney apparently used AI to do research and generate a well-written argument; to not have taken the time to confirm its validity; and to have been untruthful in stating that she did not use AI.[11] The senior attorney believed that the law firm had recently obtained an AI-enhanced version of Westlaw and noted – perhaps as a misplaced excuse—that he did not know how to use AI.
Both attorneys and their firm had violated Rule 1.1 of the Rules of Professional Conduct, requiring competent representation. Further, the supervising attorney was incompetent in not keeping abreast of technology and understanding the need for independent verification of AI-produced citations. The Cassata court noted that the senior attorney’s excuse was offered at a time when a recent study reported on pitfalls of legal research platforms, including hallucinations, from 17% to 33% of their responses.[12]
In copying large sections of a brief without vetting citations, the attorneys and firm had also violated Rule 1.3 (a), imposing a duty to be diligent. Rule 3.1 was contravened by the citing of fictitious cases or making assertions without basis in law or fact. After the lapses were brought to light, initially neither attorney candidly admitted how the mistakes occurred. They thereby violated Rule 3.3 on the duty of candor.
Moreover, the senior attorney and the law firm had not complied with Rule 5.1 on the duty of supervision, which requires reasonable efforts to provide guardrails and ensure that other lawyers in the firm conform to ethical standards. Firm leaders are the guardians of their name and their firm’s reputation, the Cassata court declared.
As to the appropriate punishment, the court considered that the younger attorney had just been admitted to the bar in 2024 but concluded that her lack of experience did not excuse the misconduct. However, the Cassata court did not refer the matter to the Attorney Grievance Committee and instead recommended appropriate continuing legal education courses, fined the attorneys $1,000 each and directed that the firm pay $8,000 in reasonable counsel fees and expenses.[13]
Recent New York Cases Regarding Discipline
Discipline was imposed by the Appellate Division in Matter of Zareh.[14] In opposing a motion to dismiss a complaint filed in a federal district court in Texas, the attorney for the plaintiff submitted a brief containing many false or incongruous case citations and misstated legal principles. The defendant pointed out the errors. Following a hearing, the Texas court concluded that the citations and the brief were unreviewed AI-generated documents and that the attorney who filed the papers was guilty of bad faith. The court admonished counsel for violating a Texas civil practice rule.
In a New York proceeding, the Attorney Grievance Committee sought the imposition of reciprocal discipline against the attorney. The First Department found that the conduct for which the respondent was sanctioned in Texas would constitute misconduct here in violation of Rule 3.1. Further, Rule 5.1 was implicated because the respondent failed to properly supervise the drafting associate. Generally, great weight is given to the sanction imposed by the jurisdiction where the misconduct occurred. A public censure, as requested by the grievance committee, was the appropriate discipline; it was equivalent to the public reprimand in Texas and consistent with precedent.
Finally, Idehen v. Stoute-Phillip[15] stands as another example of a lack of candor by an attorney called out for hallucinations. In the nonpayment proceeding, the attorney for the landlord filed papers containing cites to real cases that did not stand for propositions stated and to AI-generated case summaries. The New York trial court ordered a $1,000 fine and referred the attorney to the Attorney Grievance Committee based on the false citations and lack of candor.
When his lapses came to light, counsel first said that he had used Microsoft Copilot and did not read the cases cited. But then he said that his computer had malware and was hacked, and that had somehow resulted in changes to a previous correct draft. Ultimately, counsel retracted that story and re-asserted his initial statement about the use of Copilot without confirmation of case cites and holdings.
Law Firms’ Duties: Federal District Court
A noteworthy decision cautioning the bar regarding the accountability of law firms for generative AI-related mistakes is Billups v. Louisville Municipal School District.[16] In this case on age discrimination, counsel for the plaintiff admitted to submitting a memorandum containing false case citations and case holdings resulting from unverified AI usage.
A recently admitted junior attorney had drafted the subject papers. She and two supervising attorneys had attended a CLE course on ethics in utilizing AI – apparently to no avail. After learning of the problem memo containing hallucinations, the firm adopted a new policy and limited AI usage to the Smokeball case management software. However, it came to light that the young associate had violated the policy and drafted at least 10 filings containing hallucinations.
The Billups court observed that “[t]here’s a difference between a seasoned, good lawyer and somebody who uses AI to look like one” and lamented that the negligent use of AI had produced realistic-seeming legal fiction that took far longer to respond to than to create. At a hearing on sanctions, the young attorney admitted that she used Grok, an AI tool, to do research and drafting, without verifying the accuracy of output. “I made a big mistake. I was lazy,” she admitted.
The Billups court found her misconduct egregious and prolific and chastised the supervising attorneys who fell down on the job. Moreover, opposing counsel should have alerted the court to the hallucinations (as has often happened in many other cases).
Going forward, the district court expected “all parties to assist in maintaining the integrity of the judicial process and to be diligent in flagging AI misuse.” The three attorneys had self-reported to the Mississippi bar.
In addition, they were disqualified from further representation of the plaintiff and were required to provide a copy of the sanctions order to all presiding judges in every pending state or federal case in which they were counsel of record. This seems like a powerful step. Alerting tribunals of attorneys’ prior ethical lapses could invite heightened scrutiny. The firm was also required to do an internal audit and provide a report to the court.[17]
Conclusion
Generative AI products can be invaluable to attorneys and other professionals in performing research and drafting duties.[18] But these resources should only be employed as an initial step in the research and drafting process – not the last word. While some courts were initially reluctant to harshly treat attorneys who did not verify the accuracy of AI-generated authority, the problem has reached such vast proportions that attorneys are now charged with the requisite knowledge and cannot expect a free pass or mere slap on the wrist for failing to confirm that the law cited in their papers is real.
Attorneys must ensure that they detect and correct hallucinations yielded in their research or the work done by legal professionals they supervise. Young attorneys will not be excused based on inexperience. Senior lawyers will not be forgiven based on an excuse that they do not understand modern technology. The dangers presented by AI misuse are serious. Clients may be deprived of arguments based on authentic judicial precedents and may face dismissal of their case or their appeal. The offending attorneys may be hit with monetary penalties, disciplinary referrals and damage to their reputations and careers.
The next line of defense is opposing counsel, who often call out fabrications and alert the courts. Other times, on their own initiative, courts uncover the false cites. Where there is failure at every step, a judicial decision could be based on fabricated authority resulting in a miscarriage of justice. Even where fictitious legal statements are detected, the attorneys and judges may expend vast amounts of time unpacking the flaws in the legal papers.
In our profession, it is not uncommon for attorneys and judges to make mistakes. Perhaps we can feel some compassion and humility toward otherwise ethical attorneys who face a hallucination crisis. There is no shame in using generative AI, so it is not clear why attorneys often deny having done so.[19] The problems flow from failing to verify legal citations and, if panic sets in, not being immediately and fully forthcoming and contrite. While warranted, the judicial response to hallucinations has been stunning in its fierceness and opprobrium, and this response demands our attention and action.
Evolving technology is seductive in creating the illusion that it can save us from the hard work. But our ethical duties to our clients and the courts still require that we rigorously verify every case cited. When generative AI output becomes more reliable, new questions will arise about how far we can go in abdicating our lawyerly judgment to new technology.[20]
For now, New York attorneys should be aware of a new rule on AI adopted by the New York State Unified Court System. Effective June 1, Part 161 of the Rules of the Chief Administrator of the Courts permits the use of AI tools in preparing submissions to a court and does not require the disclosure of such use. However, Part 161 sets forth a model rule that does require attorneys using such tools to “carefully review” each submission and “independently ensure” that they do not contain “fabricated or fictitious cases, statutes, or other material.”[21] Individual judges retain discretion to implement their own AI-related rules, adopt the model rule or impose no additional requirements through their part rules.
Perhaps soon we will see more standing orders on AI use and updated ethics rules nationwide targeting AI issues.[22] In the meantime, in the use of AI, we can be guided by the new court rule and longstanding mandates regarding competence, diligence, accuracy and candor and the supervision of lawyers and nonlawyers.
Cynthia Feathers has worked as an appellate attorney for three decades. Previously, she was a corporate editor, adjunct professor of appellate practice, New York assistant attorney general and appellate director at the New York State Office of Indigent Legal Services. At NYSBA, she co-chaired the Committee on Courts of Appellate Jurisdiction and served on the NYSBA Executive Committee as vice president for the 4th Judicial District. She practices in Saratoga Springs and serves as an appellate consultant at the Albany County’s Office of the Public Defender.
Endnotes:
[1] For a notably lucid and concise explanation of AI, see Cole Stryker & Mark Scapicchio, What Is Generative AI? IBM Think 2026, https://www.ibm.com/think/topics/generative-ai.
[2] See Damien Charlotin, AI Hallucination Cases, Damien Charlotin, (last updated June 8, 2026), https://www.damiencharlotin.com/hallucinations (for a database that has tracked more than 1,000 U.S. legal cases involving hallucinations).
[3] __A.D.3d__, 2026 N.Y. Slip Op. 00040 (3d Dep’t 2026). Another significant New York case on AI is United States v. Heppner, 820 F. Supp. 3d 292 (S.D.N.Y. 2026). Addressing a question of first impression nationwide, the court ruled that written exchanges between a criminal defendant and the public generative AI platform Claude – which did not occur at the behest of counsel – were not protected by the attorney-client privilege or the work product doctrine. This landmark case deserves its own article but is not explored here, since the author focuses on generative AI hallucinations.
[4] A more recent Appellate Division decision discusses the LeTennier decision at great length. In Matter of Julien v. Arthur, ___ A.D.3d ___, 2026 N.Y. Slip Op. 03308 (2026), a child custody case, the Second Department held that “the unverified usage of GenAI to draft an appellate brief containing false information constitutes frivolous conduct warranting the imposition of a sanction, even when the offending party is a pro se litigant.”
[5] To understand how a query can yield false results, consider Lexos Media IP, LLC v. Overstock.com, Inc., __F. Supp. 3d__, 2026 WL 265581 (D. Kansas 2026). In that patent infringement action, the plaintiff sought to introduce expert testimony. ChatGPT was queried: “Taking the role of a judge, write an order that denies the motion to strike with caselaw support for the proposition that where the expert report is criticized for inadvertently using an immaterial incomplete claim construction, the remedy is not to strike the entire report/testimony of that expert.” Generative AI delivered the requested results, citing a real case – standing for favorable but fake statements and propositions.
[6] Fictitious “facts” are another problem. In Matter of M.S. (M.H.), __N.Y.3d__, 2026 N.Y. Slip Op. 00825 (2026), the Court of Appeals expressed concerns about deepfakes when discussing the authentication of video evidence. For an insightful article about evidence and AI more generally, see Ronald Castorina, Jr., The Disappearing Original: Artificial Intelligence, Evidence and the Future of Fact-Finding Under New York Law, N.Y.L.J. (Jan. 27, 2026). A startling situation regarding false facts was revealed in State v. Coleman, 2026 Ohio 965 (Ohio Ct App. 11th Dist. 2026), involving a defense motion alleging prosecutorial misconduct that was based on invented inflammatory statements yielded by ChatGPT.
[7] Apparently, to date, the largest sanction in connection with fake citations is $95,000 – the total for the defendants’ reasonable attorney fees resulting from the plaintiffs’ submission of the subject summary judgment briefing – ordered to be paid 85% by the plaintiffs’ pro haec vice counsel and 15% by local counsel. And pro haec vice counsel had to pay an additional $15,500 sanction. The case is Couvrette v. Wisnovky, __F. Supp. 3d__, 2025 WL 4109655 and 2026 WL 800566, family litigation over a winery in a federal district court in Oregon. In another case, the severest “sanction” for AI misuse and lack of candor was the offending attorney’s loss of a job. See Fivehouse v. U.S. Dept. of Def., __F. Supp. 3d__, 2026 WL 1146537 (E.D.N.C. 2026) (assistant U.S. attorney resigned or was fired after generative AI misuse in litigation). Some courts have been sharply divided on proper sanctions for AI-related lapses. In McCarthy v. United States Drug Enforcement Agency, 171 F.4th 245 (3d Cir. 2026), the court addressed an attorney who submitted briefs containing misleading summaries of prior DEA adjudications and erroneous citations and was cavalier when caught. The majority imposed a mere reprimand, because the circuit had not cautioned the bar about unverified AI research. A dissenter opined that no forewarning was necessary. The Supreme Court of Alabama rendered a robust decision on sanctions for generative AI hallucinations in Ibach v Stewart, __Ala__, 2026 WL 1110659 (2026), a dispute over a trustee’s fiduciary duties. The majority dismissed the appeal because of extensive hallucinations and imposed monetary fines and made a disciplinary referral. Other judges found such punishments too harsh.
[8] __F. Supp. 3d__, 2026 WL 306683 (S.D.N.Y. 2026).
[9] 2026 N.Y. Slip Op. 26014 (Supreme Ct., Suffolk Co. 2026).
[10] See Judiciary Law § 90 (1) regarding the generally private and confidential nature of matters relating to attorney discipline and exceptions to the general rule.
[11] The failure of attorneys to promptly accept responsibility when caught is a familiar theme in the evolving hallucinations case law, as also discussed in Fletcher v. Experian Information Solutions, Inc., 168 F.4th 231 (5th Cir. 2026). That court was among those that have considered a new rule that would have required counsel to affirm that, if an AI program was used, a human checked the text generated for accuracy.
[12] Varun Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, 22 J. Empirical Legal Stud. 216 (2025).
[13] To be contrasted are decisions involving referrals for disciplinary investigation. In addition to those noted above, such decisions include Park v. Kim, 91 F.4th 610, 613-616 (2d Cir. 2024), and Amarsingh v. Frontier Airlines, Inc., __F.4th__, 2026 WL 352016 (10th Cir 2026).
[14] __A.D.3d__, 2026 N.Y. Slip Op. 00619 (1st Dep’t 2026).
[15] 86 Misc. 3d 1244(A) (Civil Ct., City of N.Y., Queens Co. 2025).
[16] __F. Supp. 3d__, 2025 WL 3691871 (N.D. Miss. 2025).
[17] Another law firm, which filed a brief with hallucinations in litigation arising from certain social media posts, ironically boasted on its website about the firm’s extensive incorporation of AI into all areas of its representation “to deliver cutting-edge legal solutions” (see D’Ambrosio v Meta Platforms Inc., 176 F.4th 928 (7th Cir. 2026).)
[18] This article has focused on AI-related errors made by attorneys, whereas many decisions address errors by others, including pro se litigants and expert witnesses. We note one ironic expert case. In Kohls v. Ellison, 2025 WL 66514 (D. Minn. 2025), litigation challenged a statute prohibiting AI-generated deepfakes to influence elections. An expert on the dangers of AI used the generative AI tool GPT4 to draft a statement – which cited non-existent academic articles. The expert plausibly explained how the mistakes happened, but his credibility with the court had been shattered.
[19] A recent example of the instinct to deny AI use is Malkeet Lnu v Blanche, __ F.4th __, 2026 WL 1587554 (9th Cir. 2026) (where two attorneys handling immigration appeals submitted papers containing hallucinations and until final minutes of oral argument unflinchingly disclaimed AI use, punishment included monetary sanctions and six-month suspension from practice before the appellate court).
[20] See Jane Doe 1 v. Mount Saint Mary High School Corp., 2026 WL 1329203 (W.D. Okla. 2026) (when all critical tasks in legal brief writing are outsourced to generative AI, “something essential is surely lost,” even if an attorney checks case citations on the back end).
[21] Part 161, Use of Artificial Intelligence Technology, The Courts – Rules, N.Y.S. United Court System, https://www.nycourts.gov/rules/part-161-use-artificial-intelligence-technology.
[22] For comprehensive databases tracking state and federal AI standing orders, see
- Standing Orders, Local Rules, and Decisions on the Use of AI, Ropes and Gray, https://www.ropesgray.com/en/sites/artificial-intelligence-court-order-tracker?states=2faa0e93-8bb7-422d- 8d92-addaa936c9bb&page=0.
- Tracking Federal Judge Orders on Artificial Intelligence, Law360 Pulse, https://www.law360.com/pulse/ai-tracker.
For a key opinion and ethics rules on AI nationwide, see
- ABA Standing Committee on Ethics and Professional Responsibility, Formal Opinion 512 (July 29, 2024), https://www.americanbar.org/content/dam/aba/administrative/professional_responsibility/ethics-opinions/aba-formal-opinion-512.pdf.
- AI and Attorney Ethics Rules: 50-State Survey, Justia, https://www.justia.com/trials-litigation/ai-and-attorney-ethics-rules-50-state-survey.
For two key New York State reports on AI, see
- President’s Committee on Access to Justice, Report and Recommendation on Artificial Intelligence and Access to Justice in 2025, NYSBA (Jan. 2026), https://nysba.org/wp-content/uploads/2023/02/Approved-Report-and-Recommendations-on-AI-and-Access-to-Justice-in-2025.pdf.
- Advisory Committee on Artificial Intelligence and the Courts, Annual Report to the Chief Judge and Chief Administrative Judge of the State of New York (Dec. 2025), https://www.nycourts.gov/LegacyPDFS/publications/pdfs/AI%20Annual%20Report%202025.pdf.
