The Last Independent Number: Legal AI Accuracy Two and a Half Years After Stanford
Current as of October 12, 2026. Where sources disagree, the text says so. Disclosure: I build AI tools for my own firm and use several of the products named in this article. This article is educational and is not legal advice.
The question this article answers: Have Westlaw, Lexis, Harvey, and the other legal AI tools become more accurate since Stanford's 2024 hallucination study, what problems are lawyers and firms actually having with them, and what should the profession do about it?
Bottom Line Up Front
The underlying AI models fabricate far less law than they did in 2023. The research platforms built on them have probably improved too, but no one outside the vendors has measured it, and the two largest vendors have not appeared in any public benchmark of legal research accuracy since. The last independent hallucination rates for the platforms most U.S. lawyers use are still Stanford's from 2024: about 17% for Lexis+ AI and about 33% for Westlaw's AI-Assisted Research (Magesh et al., Journal of Empirical Legal Studies).
Meanwhile, the error has changed shape, and the consequences have grown. When the newest benchmarks grade legal work the way a supervising partner does, where one missed requirement sends the work back, the best models complete roughly one task in four. When a hallucination check is added, the best score falls below one in ten, and more than 60% of otherwise passing work turns out to contain a material hallucination such as a wrong party, amount, date, or obligation (Artificial Analysis, Harvey LAB-AA v1.1). In the courts, Damien Charlotin's database of decisions addressing AI-hallucinated material grew from about 120 entries in May 2025 to 2,184 on October 10, 2026 (Charlotin). Error rates at the bottom of the stack fell. Consequences at the top of the stack rose. Both facts are true, and a firm that plans around only one of them will get hurt by the other.
Six conclusions follow, and they organize everything below.
First, a hallucination rate is a property of the test as much as the product. The definition, the question set, and the yardstick used to check the answers move the number by more than product quality does. In one 2026 benchmark, six different AI "checkers" reviewing the same deliverables upheld anywhere from 28 to 470 material hallucinations. Any accuracy figure offered without its definition, questions, and checker is marketing.
Second, the raw models improved sharply on fabrication and only modestly on everything else. They still defer to a confident user, still accept a real case cited for the wrong page, and still regress from one release to the next.
Third, strict grading exposes a wall. Models that satisfy 94% of individual criteria complete only 20% to 25% of long legal tasks without a miss, and long work products are where firms most want to use them.
Fourth, the most dangerous errors are no longer fake cases. They are real cases cited for propositions they do not support, invented facts about your own record, and legal judgment that is simply wrong while looking like boilerplate. Citators and automated checkers are built for the first category and miss the others.
Fifth, legal-specific wrappers such as Harvey, CoCounsel, Protégé, and Legora earn their price through licensed content, citators, connectors, and workflow, not through better reasoning. Independent tests put specialized and general tools within a point or two of each other, and the vendors' own data, read carefully, says the same thing.
Sixth, lawyers using AI are faster and produce more polished work, but not measurably more accurate work, and the junior lawyers who lean on it most appear to retain the least. Trust has to be built into the workflow. It cannot be purchased with the license.
This article proceeds in eleven parts. Part I explains why "have the percentages gone down?" is a trick question. Part II revisits the Stanford baseline and the silence that followed it. Part III measures the raw models. Part IV describes the strict-grading wall. Part V explains how the error has changed shape, including a patent-specific failure mode every litigator should know. Part VI asks what a wrapper like Harvey actually buys. Part VII reviews the court record. Part VIII looks at lawyers using AI, measured in randomized trials and surveys. Part IX gives the vendors' strongest case a fair hearing and explains why it does not change the answer. Part X explains why the market has not produced the number. After a layer-by-layer answer and short answers to common questions, Part XI closes with recommendations for lawyers, patent practitioners, firm leaders, clients, vendors, and courts.
Over the past week I published five short posts on this subject, one layer of the stack per day. This article draws on them but does not repeat them. It adds the detail, the counterarguments, and the sources the posts could not hold.
I. Why "Have the Percentages Gone Down?" Is a Trick Question
Three things make the question harder than it looks, and each one explains part of why the vendors and their critics talk past each other.
A. "Hallucination" means at least two different things
The narrow meaning is fabrication: the cited case, statute, or quotation does not exist. The broad meaning, which Stanford used, adds misgrounding: the source exists, but it does not support the proposition it is cited for. When LexisNexis promised "hallucination-free linked legal citations," it was promising the first. When Stanford reported 17%, it was measuring both (LawSites). Both statements can be true of the same product on the same day. The dispute has never been resolved because nobody has run a test on the current platforms that reports the two numbers separately.
The distinction matters more now than it did in 2024, because the two kinds of error have moved in opposite directions. Fabrication is the error that retrieval grounding, linked citations, and citators are designed to catch, and they catch most of it. Misgrounding is the error that survives all three. A brief that cites a real Federal Circuit opinion for a holding it never reached passes every existence check ever built.
B. The yardstick moves the number
A 2026 study of Ukrainian legal questions scored the same answers from four commercial models against a sparse reference database and a dense one. Against the sparse reference, 15% to 21% of citations appeared to be hallucinated. Against the dense reference, 98.9% to 99.9% of the same citations were grounded. All 54 citations the sparse database flagged turned out to be real, a 100% false-positive rate (Ovcharov, Citation Grounding Measures the Oracle). The models did not change. The reference did.
The same thing happens when AI checks AI. When Artificial Analysis added a hallucination audit to Harvey's Legal Agent Benchmark this month, it first compared six candidate checker models on the same 20 tasks and the same deliverables from eight models. The most aggressive checker upheld 470 material hallucinations. The least aggressive upheld 28. A frontier model from one lab upheld 99; a smaller model from the same lab upheld 57 (Artificial Analysis). The deliverables were identical. The count varied by a factor of almost seventeen depending on who did the counting.
Any vendor that reports a citation-validity rate owes you three facts: the database it checked against, the version of the product it tested, and whether a human or a model did the checking. Without them, the figure cannot be compared to anything, including the vendor's own figure from last year.
C. The stack has layers, and they moved in opposite directions
A frontier model, a research platform built on licensed content, a workflow wrapper like Harvey, an agent completing a twenty-step assignment, and a lawyer using any of them are five different things. The evidence for each is different, and the honest answer is different for each. Courts are a sixth layer: they do not measure accuracy at all, but they measure consequences, and that measurement has moved fastest.
Bottom line: before you accept or reject any accuracy claim, ask what it counts, who wrote the questions, and what checked the answers.
II. The Stanford Baseline, and the Silence That Followed
Stanford RegLab and the Institute for Human-Centered AI built a preregistered set of 202 queries in four categories, including 22 false-premise questions and 70 jurisdiction- or time-specific questions. They ran Lexis+ AI, Ask Practical Law AI, and GPT-4 in the spring of 2024, and added Westlaw's AI-Assisted Research in a later round of testing. Human reviewers graded each response for correctness and for whether the cited sources actually supported the propositions (Magesh et al.; Stanford HAI).
Lexis+ AI answered accurately about 65% of the time and hallucinated on about 17% of queries. Westlaw's AI-Assisted Research was accurate about 41% to 42% of the time (the paper reports both figures in different sections) and hallucinated on about one query in three. Ask Practical Law AI hallucinated on about 17% but gave incomplete answers 62% of the time. GPT-4, with no legal database at all, hallucinated on 43%.
Two of the paper's observations matter more now than they did then. Westlaw's answers averaged 350 words against Lexis's 219, and the authors explained that longer answers contain more falsifiable propositions and therefore a greater chance of at least one hallucination. Longer answers also take longer to check, because every proposition and citation has to be independently evaluated. That is the first statement of the arithmetic that now governs agentic legal AI, discussed in Part IV.
A. The vendors' response: process, not percentages
LexisNexis responded that "the study authors have taken a wider position on what constitutes a hallucination" and restated its promise of "hallucination-free linked legal citations" (LawSites). Thomson Reuters said its "thorough internal testing of AI-Assisted Research shows an accuracy rate of approximately 90% based on how our customers use it" (Artificial Lawyer). Neither the questions, the rubric, nor the definition of accuracy behind that 90% has been published.
Since then the vendors have described process rather than results. In May 2025, LexisNexis explained that each Protégé prompt "is checked across five stages" and that a citation that cannot be linked is flagged "for manual validation," without attaching an error rate (LexisNexis). In January 2026, the Thomson Reuters Institute published a 22-page report on managing hallucinations in courts. It said "AI must serve as an assistant, not an arbiter" and recommended treating verification "as a workflow requirement, not an afterthought," but it reported no accuracy figure for any Thomson Reuters product (Thomson Reuters Institute).
For whatever one lawyer's experience is worth, a little over a year ago I ran a trial of CoCounsel, and on my matters it performed well below the free version of ChatGPT.
B. The withdrawals
Vals AI's first legal benchmark, published in February 2025 with questions and answer keys contributed by law firms, evaluated Harvey, CoCounsel, Vincent AI, and Oliver across seven tasks. The report states that "LexisNexis chose to withdraw from all skills except legal research and, therefore, does not appear in this report" (Vals Legal AI Report). When Vals published its legal research study in October 2025, neither Thomson Reuters nor LexisNexis participated, and vLex withdrew. The products tested were Alexi, Counsel Stack, Midpage, and ChatGPT (Vals; Legal IT Insider).
C. What later tests of the platforms do show
No one has rerun the Stanford protocol on the current products. Several later studies touched them, and none is reassuring.
A peer-reviewed study that ran 50 legal questions through several systems between December 2024 and April 2025 classified Lexis+ AI's answers as 58% accurate, 22% incomplete, and 20% hallucinated. That was better than ChatGPT-4 (36% hallucinated), Copilot, DeepSeek, or Llama 3 in the same test, but it is not evidence of improvement over Stanford's 17%, because the questions and coding differed and the sample was small (Munir et al., International Journal of Legal Information).
In 2026, Stanford's Daniel Ho and colleagues tested the statutory-survey features of both platforms on 1,647 questions about state unemployment insurance law, using a Department of Labor survey as the answer key. Westlaw's tool reached 58% accuracy and Lexis+ AI's reached 64%, both "even worse than standard RAG," a basic retrieval baseline that scored 70%. Lexis found only 29% of the provisions that actually existed. A purpose-built academic system scored 83%, rising to 92% once the researchers discovered that the government's own answer key had omitted valid statutes (Afane et al., Benchmarking Legal RAG). That last detail is worth pausing on: the human gold standard was itself wrong often enough to move the score nine points.
Outside the United States, a University of British Columbia law professor gave the Canadian version of Lexis+ AI "a failing grade" after it cited a "Section 15.07 of the Canada Legislation" that does not exist (CBA National). A 2025 panel of law librarians who put the same California questions to Lexis+ AI, Westlaw Precision AI, and Vincent AI found that the tools split on whether a statute created a private right of action and that one omitted a controlling doctrine entirely (LawSites). And a large firm's own trials found something the vendors rarely mention: when Ashurst tested several tools in 2024, its lawyers reported "frequent hallucinations across all tools trialled," and human work scored higher on accuracy (Legal Futures).
None of these is a hallucination rate in Stanford's sense. All of them say the same thing: the platforms are useful, they are not reliable enough to sign, and nobody outside the vendors knows exactly how reliable they are.
The vendors' best argument is that their products change monthly, so a 2024 test describes a product that no longer exists. That is true. It is also the strongest argument for continuous independent testing, not against it. Stanford's dataset is public, with a random half reserved as a clean test set. Running it again requires access to the current products and careful grading, and nothing else. In more than two years, no one with access to the current products has published the result.
Bottom line: the 17% and 33% figures describe 2024 products, and they remain the last independent numbers because the vendors have chosen not to replace them.
III. The Raw Models Improved, With Four Asterisks
Here the evidence is unambiguous, and lawyers should not pretend otherwise. In a study published in 2024, Stanford's "Large Legal Fictions" project asked general-purpose models direct, verifiable questions about randomly selected federal cases and found that they "hallucinate between 58% (ChatGPT 4) and 88% (Llama 2) of the time." The authors added that the models "struggle to predict their own hallucinations, and often uncritically accept users' incorrect legal assumptions," and that they perform better on newer, more prominent cases from more prominent jurisdictions (Dahl et al., Journal of Legal Analysis). Since then, three independent lines of evidence agree that fabrication has fallen steeply.
Linklaters put 50 hard English law questions across ten practice areas to the 2023 models and found that "in a third of cases (47 out of 150 answers)" the models relied on case law or guidance that was fictitious. In the firm's February 2025 rerun, OpenAI o1 scored 6.4 out of 10 and Gemini 2.0 scored 6.0, against Bard's 4.4 the year before, with "fictional citations only being returned in 9% of queries" (Linklaters, LinksAI Version 2). OpenAI's GPT-5 system card reports that on production traffic, the share of responses containing at least one major factual error fell from 22.0% for o3 to 4.8% for GPT-5 thinking, a self-reported figure (OpenAI). And on LegalBench, the open legal-reasoning benchmark, the top score rose from 77.7% in April 2024 to 88.56% today, a level Vals AI describes as "largely saturated: the top models are bunched near 88%" (Vals AI; The Batch).
Now the asterisks, because each one maps onto a sanctions case in Part VII.
A. Fabrication fell. It did not vanish.
Linklaters' improved Gemini 2.0 cited "UBS AG v Rose Capital Ventures Ltd [2019] EWCA Civ 2128" for a proposition about reasonable endeavours. The case does not exist. The firm's conclusion did not change across both rounds: the models "should not be used for English law legal advice without expert human supervision," and "there are real risks to using them if you don't already know the answer" (Linklaters). Nine percent of answers is not a rounding error. A practice group that asks the tool a hundred research questions a month would, at that rate, receive about nine answers resting on authority that does not exist.
B. Progress is not a straight line.
OpenAI's own system card for o3 and o4-mini, released in April 2025, showed the newer models hallucinating more than their predecessor on the PersonQA test: 33% for o3 and 48% for o4-mini, against 16% for o1 (OpenAI). The same pattern appears in legal benchmarks. On Vals' CaseLaw v2 benchmark, which asked questions about supplied Canadian court decisions, the older GPT-4.1 outscored the newer GPT-5, 69.88% to 66.45% (Vals AI). A firm that validated a workflow on one model version has not validated it on the next.
C. The models still defer to the user.
Stanford's 2026 AI Index reports that when a false belief is framed as the user's own, GPT-4o's accuracy fell from 98.2% to 64.4%, and DeepSeek R1's fell from over 90% to 14.4% (Stanford HAI, 2026 AI Index). Stanford's 2024 legal study had already built in 22 false-premise questions for exactly this reason, and an August 2026 study of eight research-built legal retrieval pipelines, using GDPR and French civil law, again found false-premise questions to be the hardest category on a small sample, with hallucination rates ranging from under 10% for the best configuration on civil law to 48.3% for the worst (Das, Abualhaija & Bianculli). The practical lesson is uncomfortable. A lawyer who types a conclusion and asks the tool to find support for it is running a false-premise test on every query.
D. Knowing the law is not the same as reading the record.
Vals' explanation of the CaseLaw results is the most important sentence in that benchmark: a common issue was models "relying more on their general knowledge rather than the specific document context (despite being instructed otherwise)" (Vals AI). That is the model that knows the general rule and ignores your record. Vals retired the benchmark as saturated after later models improved, but the failure mode is structural. A model trained on the whole of the internet will always be tempted to answer the question it expected instead of the question your documents pose.
Bottom line: the models are much better at not inventing law. They are not much better at resisting a confident user, at reading the record in front of them, or at staying consistent from one release to the next.
IV. The Strict-Grading Wall
The most important development in legal AI evaluation since Stanford is not a new product. It is a new grading standard. The 2026 benchmarks score a deliverable against a rubric of specific, binary criteria, and the deliverable passes only if every criterion passes. That is how a partner reviews an associate's draft, and it is how a court reviews a brief. Under that standard, the same models that saturate LegalBench fail most long legal tasks.
A. Harvey's Legal Agent Benchmark
Harvey released its Legal Agent Benchmark (LAB) in May 2026: more than 1,200 tasks across 24 practice areas, graded against more than 75,000 rubric criteria, with agents working through documents, spreadsheets, and file tools (LawSites). Harvey's own initial results were candid: "the frontier models we evaluated complete less than 10% of tasks end-to-end in aggregate," and the top performer cost "roughly $50 per task" with "over 20 minutes of latency" (Harvey).
Vals now runs LAB on a held-out set. As of this week, the leading models satisfy about 94% to 95% of individual criteria but complete only 20% to 25% of tasks with every criterion met. The top model completes 25.42%. Claude Fable 5 completes 11.25%, Claude Opus 5 6.67%, GPT-6 Astra 5.42%, and GPT 5.5 3.75%. In Vals' words, "Harvey grades a task as resolved only if every criterion passes. A model can satisfy most individual criteria and still miss task resolution credit" (Vals AI).
B. Add a hallucination check, and the best score falls below 10%
On October 8, 2026, Artificial Analysis, working with Harvey, released version 1.1 of its independent implementation of the benchmark. Every deliverable is now audited against the task's own source documents, and a task earns no credit if it contains a single material hallucination, which the evaluators define as a claim that "would mislead a reader on a substantive point - a wrong party, amount, date or obligation, or a fact that changes a conclusion." The best model scores 9.4%. Without the hallucination gate, the leader would have scored 26.7%, "but two thirds of those passes contain a material hallucination." Across the models tested, "more than 60% of otherwise passing results contain a material hallucination, and several models score 0% after accounting for hallucinations" (Artificial Analysis).
Two findings from that release deserve more attention than they will get. First, completing the work and not hallucinating are different skills. The model that passed the most rubric criteria, 96.0%, averaged 1.68 material hallucinations per task, while the model with the fewest hallucinations averaged 0.03 per task and still passed fewer than one task in ten outright. Second, the hallucinations were not exotic. They were wrong numbers, wrong dates, and statements that something had happened when the source said it still needed to be confirmed. Those are the errors that end up in a closing memo, a damages model, or a declaration.
C. Legal research, graded strictly
Vals' Legal Research Bench uses 208 held-out research questions. Claude Opus 5 earns 90.58% of weighted rubric points but answers only 55.29% of questions entirely correctly, tied for first. Vals explains why it reports the strict number: "in legal work, a partially correct answer can be more dangerous than a wrong one: it may read as sound while omitting a critical point" (Vals AI). On that same board, OpenAI's new GPT-6 Astra model, used without OpenAI's legal search index, scores 39.42%.
DELTA, a public benchmark of Dutch legal research built from practitioner assignments, tested 23 models. The best passed 15.0% of tasks while satisfying 75.7% of criteria, and the benchmark's conclusion is blunt: "No model is ready to conduct Dutch legal research independently" (Legal Benchmarks, DELTA).
D. The arithmetic, and the one hopeful finding
None of this is mysterious. If a deliverable contains fifty checkable claims and each is independently 95% likely to be right, the chance that all fifty are right is about 8%. At 99% per claim, it is about 61%. Errors in real models are correlated, so they do better than the independent-error math, but the principle holds: reliability per step does not translate into reliability per deliverable, and the longer the deliverable, the worse the translation. Stanford saw the early version of this in 2024, when the tool with the longest answers had the highest hallucination rate.
The hopeful finding is in Harvey's own analysis of what separates passing from failing runs. Agents that added a validation step after drafting scored about 0.8 points higher, agents that revised after checking scored about 1.5 points higher, and agents that drafted without later validation scored about 1.2 points lower (Harvey). The machines are learning to cite-check their own work. The lesson for lawyers is the same one every senior associate learned the hard way: the draft is not the product. The checked draft is.
Bottom line: assume every long AI work product contains at least one error until someone has ruled it out. On the current evidence, that assumption is correct more often than not, even for the best models.
V. The Error Has Changed Shape
The fabricated case is now the easier error to catch. Retrieval grounding, linked citations, citators, and models asked to check their own work catch most of them. Three harder errors have taken its place.
A. Real cases, wrong support
A 2026 study presented at the ICML AI for Law workshop tested fourteen model configurations on controlled citation corruptions. "Models catch 93-100% of wrong-case corruptions. They catch only 37-61% of wrong-pinpoint corruptions on court opinions and 52-83% on legal briefs." Even GPT-5.4 at high reasoning effort "still misses 40% of pinpoint mismatches on court opinions." The reason is the one any appellate clerk would predict: the models "accept the citation based on topical overlap rather than page-level support" (Verma, Is this Citation on Point?).
That finding reframes the "use AI to check the AI" strategy. Princeton researchers who assembled more than 1,000 real filings containing fabricated citations built a benchmark of 1,300 brief excerpts with injected errors. GPT-5, working as an agent, reached 84.4% recall but only 40.8% precision, for an F1 score of 55.0%, and the best system tested reached 68.8% F1 (Liu, Stammbach & Henderson, Who Checks the Citations?). The companion post reports that GPT-5 "catches only 18.2% of incorrect pincites" (Princeton CITP). Automated cite-checking is a reasonable first filter. It is not a substitute for a human reading the cited page, because the errors it misses are exactly the ones a court will notice, and the false alarms it raises cost reviewer time of their own.
B. Real record, invented facts
The second shift is from invented law to invented facts. The LAB hallucination audit described in Part IV did not look for fake cases. It looked for statements that contradicted the source documents, fabricated their content, or asserted specifics the record did not support. Those are the errors that matter in transactional work, in damages, and in patent litigation, where the most important citations are often not to case law at all.
On July 27, 2026, the USPTO's Office of Enrollment and Discipline publicly reprimanded a registered practitioner for AI-generated errors in a joint claim construction chart filed in a patent case in the Eastern District of Pennsylvania. The errors were not citations to case law. They were citations to the patent's own specification and prosecution history. The order warns that "the potential pitfalls of AI-generated citation errors are not limited to extrinsic sources such as statutes, regulations, and case law, but may also extend to the intrinsic evidence" (In re Mitchell, USPTO OED Proceeding No. D2026-16). According to opposing counsel's count, 47 quotations in the chart did not exist, 26 of them attributed to the file history. The practitioner had used one AI tool to pull the citations and a second AI tool to verify them (IPWatchdog).
Every patent litigator should read that order twice. Claim construction briefs, infringement and invalidity contentions, expert reports, and claim charts are built on column-and-line citations to the intrinsic record and pinpoint citations to product documentation. A model that is 95% reliable per citation will, across a 200-row claim chart, produce errors that look exactly like real evidence. And a second model checking the first is not independent verification. It is a second draw from the same kind of process.
The USPTO said as much in 2024: "Simply relying on the accuracy of an AI tool is not a reasonable inquiry" (USPTO Guidance, 89 Fed. Reg. 25609). The research literature on patent tasks gives the same warning from another direction. In PANORAMA, a NeurIPS 2025 benchmark built from 8,143 U.S. examination records, the best model classified novelty and nonobviousness correctly less than half the time, only modestly better than a random baseline (Lim et al., PANORAMA). Unreviewed AI text has even reached the patent record itself: Australian application AU2023233168A1, published in March 2025, contains in its background section the sentence "As of my last knowledge update in September 2021, I do not have access to specific patent numbers" (Google Patents).
C. The error that no checker can catch
The most dangerous error I have seen this year was not a citation at all. I recently asked Harvey, ChatGPT, and Claude to draft the same document: a standstill agreement that would let a patent owner open discussions with an accused infringer without creating declaratory judgment jurisdiction.
The stakes of that drafting exercise are set by two cases every patent litigator knows. Under MedImmune, Inc. v. Genentech, Inc., 549 U.S. 118, 127 (2007), the question is whether the facts show "a substantial controversy, between parties having adverse legal interests, of sufficient immediacy and reality to warrant the issuance of a declaratory judgment" (quoting Maryland Casualty Co. v. Pacific Coal & Oil Co., 312 U.S. 270, 273 (1941)). And under SanDisk Corp. v. STMicroelectronics, Inc., 480 F.3d 1372, 1381 (Fed. Cir. 2007), jurisdiction arises "where a patentee asserts rights under a patent based on certain identified ongoing or planned activity of another party, and where that party contends that it has the right to engage in the accused activity without license." The entire purpose of a standstill agreement is to keep the patent owner on the right side of that line.
Harvey's draft piled on whereas clauses that added nothing, and one that was worse than nothing: a recital that Party X had accused Party Y of infringing patents A, B, and C. Presenting that draft to the other side would have supplied, in the patent owner's own words, the assertion of rights that creates the controversy the agreement existed to avoid. Every citation in the draft could have been real. No citator, no linked-citation check, and no automated hallucination detector would have flagged it, because nothing in it was false. It was simply wrong, and it looked like boilerplate.
That is the category of error the strict-grading benchmarks are beginning to measure and the fabrication-focused safeguards are not built to see. It is why the reviewer has to be someone who knows what the document is for.
Bottom line: check pinpoints, propositions, and facts against the source, not just whether the authority exists, and do not mistake a second model for a second reviewer.
VI. What a Wrapper Buys You, and What Firms Are Actually Experiencing
Harvey, CoCounsel, Protégé, Legora, and their competitors sit on top of models built by OpenAI, Anthropic, Google, and others. The buyer's question is what the layer in between adds. The evidence says: licensed content, citators, connectors, and workflow, yes; better legal reasoning, not measurably.
A. The independent benchmarks
In Vals' February 2025 study, Harvey entered six of seven tasks and had the top AI score on five of them, including 94.8% on document question-answering against a lawyer baseline of 70.1%. Lawyers beat every tool on redlining, and CoCounsel led summarization at 77.2% (Vals Legal AI Report). That study had no general-purpose model as a control, and it remains the only independent public benchmark ever to score Harvey.
Vals' October 2025 legal research study did include a control. The legal-specific tools scored 78% to 81% on accuracy, ChatGPT scored 80%, and the lawyer baseline scored 71%. On the authoritativeness of sources, the legal tools averaged 76% against ChatGPT's 70%, and scores on multi-jurisdiction questions ran 14 points lower across the board (Vals; LawSites). Licensed sources bought a modest edge in citing good authority. They did not buy better answers.
Legal Benchmarks, an independent evaluator founded by former BigLaw associate Anna Guo, ran 13 AI tools and a panel of human lawyers through 30 real contract-drafting tasks. Legal-specific tools were reliable on 57.6% of tasks, general-purpose tools on 58.3%, and the human lawyers on 56.7%. The one place the legal tools clearly stood out was risk-flagging: in high-stakes scenarios, 83% of their outputs raised an explicit warning, against 55% for general tools and none from the humans (LawSites). Guo's summary this May was direct: "the 'purpose-built is automatically better' claim does not survive contact with independent testing" (Legal Benchmarks).
B. The vendors' own data, read carefully
The vendors' own disclosures, read closely, tell the same story, which is to their credit.
Harvey's February 2026 post on its internal evaluations reported that, in its SEC EDGAR research product, lawyers preferred Harvey over GPT-5 with web search 51.73% of the time, a statistical coin flip, until Harvey rebuilt its data connector; afterward the preference rate was 76.7%. In its Vault product, the share of valid citations ran from 62% on one base model to 97% on another (Harvey). The edge came from the plumbing, and it moved with the model underneath.
Thomson Reuters launched its own model, called Thomson, on August 24, 2026, and reported that "Thomson working over Westlaw and Practical Law scored 0.83 on factuality against 0.65 and 0.68 for leading frontier models given unrestricted access to the open web" (Thomson Reuters). That comparison gives Thomson the Westlaw content and denies it to the rivals. As The Decoder reported from the same data, with web access alone Thomson scored 0.53 against GPT 5.4's 0.65, and when both had Thomson Reuters content, Thomson edged GPT 5.4 by 0.83 to 0.82 (The Decoder). The content did nearly all the work.
OpenAI's Astra for Law, released September 17, 2026, makes the same point from the other direction. OpenAI reports that GPT-6 Astra with its new legal search index "passed the evaluation's overall correctness check on 54.0% of questions, compared with 38.7% for GPT-6 Astra using web search alone," on questions from the validation set of Vals' legal research benchmark (OpenAI). That is a self-reported figure, but the direction is the lesson: a fifteen-point gain from better sources, on the same model.
Harvey's own help center is admirably candid about limits. Its LexisNexis integration "is a paid add-on feature within Harvey," and "existing LexisNexis customers must still buy the add-on separately." The same page lists use cases "which may produce less reliable results," including "Exhaustive 'scorched earth' legal research" (Harvey Help Center). For a litigation team, exhaustive research is not an edge case. It is the job.
C. What the adoption evidence shows
What firms actually experience is more mixed than either the demos or the critics suggest. Harvey announced a $550 million round at a $15.5 billion valuation on September 9, 2026, and says that "80% of Am Law 100 law firms use Harvey" (Harvey; Reuters). The International Legal Technology Association's 2026 survey of more than 500 firms found Microsoft 365 Copilot in use at 76% of firms, CoCounsel and Anthropic's Claude at 44% each, Harvey at 43%, Lexis+ AI at 26%, and Legora at 16% (LawSites).
Licenses and use are different things. A Harvey-commissioned study by RSGI of 40 customers reported that 92% of licenses were used monthly and that power users saved 34.1 hours a month against 14.5 for standard users, but law-firm participants were "more equivocal about Harvey's ability to drive cost savings (52 per cent)," and Legal IT Insider's follow-up noted that the "time savings are mostly estimates" (Legal IT Insider; follow-up). A second Harvey-commissioned RSGI report in June 2026 found that 48% of in-house teams required their law firms to use AI, but "only one reported receiving cost savings as a result" (Legal IT Insider). Harvard's Center on the Legal Profession, drawing on an analysis by Harvey investor Andreessen Horowitz, describes the product as "there and available" at major firms but "not heavily used yet," and observes that "adoption in terms of licenses is reportedly high, but actual use apparently remains relatively low" (Harvard CLP).
The anecdotes run the same way. A large-firm innovation director told the research firm Sacra that "Legora has better knowledge vaults than Harvey," and Sacra's own summary of the interview attributes Harvey's large-firm wins to "client pressure and brand recognition rather than product superiority" (Sacra). Guo reports that "Several lawyers have shared that their enterprise legal ai tools sometimes perform worse than ChatGPT," that some buyers get "white-glove service, others cannot get an email response," and that "Buying decisions can be driven as much by data security, integrations, and existing contracts as by functionality" (Legal Benchmarks). When a purported former Harvey employee claimed on Reddit in 2025 that lawyers were not really using the product, Harvey's CEO answered publicly with 98% gross revenue retention and 77% seat utilization (Business Insider). Both things can be true: high license counts and concentrated real use. As Artificial Lawyer's Richard Tromans put it on October 12, "Most major law firms use legal AI tools, yet there has been little significant change to how those firms operate" (Artificial Lawyer).
Thomson Reuters' products draw similar notes. A review in the ABA's GPSolo magazine this year found that CoCounsel "initially missed a critical exclusivity clause" in a test document, in a publication of an ABA division that Thomson Reuters sponsors (ABA GPSolo).
D. The first court record linking Harvey to a defective filing
On September 22, 2026, Judge Arun Subramanian of the Southern District of New York ordered counsel in Hill v. Foundation Media to explain whether certain quotations and a citation in their filing came from "unchecked use of AI." On September 29, he declined to impose sanctions, called the episode "a wakeup call," and warned that "over-reliance on AI risks stunting the training of young lawyers." On October 1, the firm disclosed that it had used Harvey to help prepare the filing (Reuters). Harvey declined to comment, and nothing in the record says which tool produced which error. But it is the first court record I have found that ties Harvey to a filing a judge had to question. It joins a growing list of court records involving legal-specific products, discussed in Part VII, and it illustrates the one risk a wrapper cannot remove: the risk that lives in the review.
E. What my own team has seen
I should be clear about the evidentiary weight of what follows. These are the experiences of one litigation team, not a benchmark. They are also the comparison that no vendor has published.
Several lawyers on my team have watched Harvey return erroneous information and hallucinated authority on a regular basis, including when its answers were grounded in LexisNexis content. Grounding in a licensed database is supposed to prevent exactly that. In our experience, it has not. The CoCounsel trial described in Part II ended the same way, with a specialized product performing well below a free general-purpose model on the work we actually do.
The standstill agreement described in Part V was my most recent side-by-side test, and it is the one I would put in front of a procurement committee. Three systems, one prompt, one deliverable that a patent litigator can grade in ten minutes. The wrapper was the one that got it wrong, and it got it wrong in the place a wrapper is supposed to add value: the legal framing around the operative terms. That is consistent with the Legal Benchmarks finding that specialized tools do not outperform general ones on substance, and with Harvey's own data showing that its edge moves with sources and connectors rather than reasoning. It is also a reminder that a wrapper's additions, the templates, recitals, and house style layered over the model, are themselves a source of error. More text is more surface area, which is Stanford's 2024 finding about long answers in a new setting.
F. The market moved underneath the buyers
Four developments changed what a wrapper is. In June 2025, LexisNexis and Harvey announced an alliance to bring Lexis primary law and Shepard's Citations inside Harvey (LawSites); the layers began to merge. On February 3, 2026, after Anthropic released a legal plugin for Claude, shares of Thomson Reuters fell 14.2%, RELX 14%, and Wolters Kluwer 10.5% (Canadian Lawyer); the market priced in the possibility that the labs could go around the content owners. In August, Thomson Reuters launched its own model; the content owner moved upstream. And in September, OpenAI began selling Astra for Law directly to selected firms, while telling customers that Harvey and Legora would build on it; the lab moved downstream (OpenAI).
Harvey, for its part, now routes work across models from several labs and, in August, released Tenet, "our first post-trained open-weight model," built on Moonshot AI's Kimi K3 (Harvey; The Next Web). Its partner Fireworks reports that the post-training lifted Kimi K3's score on Harvey's own benchmark from 10.8% to 19.7% (Fireworks), a vendor-reported result that does not yet appear on Vals' independent board. That architecture only makes sense if the model is interchangeable and the layer around it is the product. For a buyer, it means the four things bundled into a legal AI subscription, model access, licensed content, workflow, and verification tooling, should be priced and tested separately, because each is now available separately. It also means a firm should know which base model touched its client's data, and on whose infrastructure.
G. Every self-run benchmark crowns its sponsor
In October 2024, Harvey's own BigLaw Bench reported sentence-level hallucination rates of 0.2% for Harvey against 0.7% for Claude, 1.3% for ChatGPT, and 1.9% for Gemini (Harvey). Two years later, Harvey's far more demanding LAB shows every frontier model failing most tasks. Both can be accurate, because they measure different things, and that is the point. Ivo's benchmark ranks Ivo first. Paxton reports a 94.7% non-hallucination rate on a sample built from Stanford's earlier dataset, without saying who graded it. An independent critique of Legora's own benchmark said it "reads more like a sophisticated vendor-led product evaluation" (Legal Benchmarks). As Guo puts it, "these are vendor benchmarks, and vendor benchmarks tend to make the vendor look good" (Legal Benchmarks). The right discount is not zero. It is also not the vendor's.
Bottom line: buy a wrapper for licensed law, citators, connectors, and workflow, and test it on exactly those things. Then run the same matters through the product and through the model underneath it. If the product cannot beat its own base model on your work, you are paying for packaging.
VII. The Court Record Went the Other Way
A. The numbers, and what they do and do not measure
Damien Charlotin's database of judicial decisions addressing AI-hallucinated material listed about 120 cases in May 2025 (Business Insider). On October 10, 2026, it listed 2,184: 1,259 involving self-represented litigants, 864 involving lawyers, 32 involving judges, and 15 involving expert witnesses, across more than 40 jurisdictions, with 1,492 in the United States, 233 in Canada, 112 in Australia, and 76 in the United Kingdom (Charlotin). An independent analysis of the October 4 export found that decisions dated January through September 2026 numbered 1,212, about 2.7 times the 449 for the same months of 2025; that 82% of entries involved at least one fabricated item; and that 27% recorded a warning, 18% a monetary penalty, and 8% a professional sanction (Stewie). Reuters, counting state and federal courts in the United States, put the figure at "at least 1,395 cases" in September and reported three incidents in three days, including a murder appeal in which a defense lawyer "cited fabricated witnesses and made-up police testimony" (Reuters).
The count must be read carefully. It measures adoption and detection, not an error rate, because the denominator of AI-assisted filings is unknown and most filings that contain errors are presumably never flagged. More than half the entries involve people without lawyers. Among lawyers, the problem is concentrated in small practices: of 129 law firms in a 2025 Stanford analysis of lawyer cases, 90% were solo practices or small firms (Stanford CIS). And most entries never identify the tool at all. But the cases involving lawyers include some of the most sophisticated firms in the country, and the pattern in those cases is what matters for practice.
B. Who bears the risk
Case | Court and date | What happened | Consequence |
United States v. Farris | 6th Cir., Apr. 3, 2026 (published) | Principal brief saved as "CoCounsel Skill Results"; three quotations attributed to two real cases and Guidelines commentary did not appear in those sources | No appointed-counsel compensation for the appeal; referral for possible discipline; counsel replaced |
In re Prince Global Holdings (Sullivan & Cromwell) | Bankr. S.D.N.Y., Apr. 2026 | Inaccurate AI-generated citations in a motion; firm policies not followed; review missed them; opposing counsel flagged them | Written apology to Chief Judge Glenn |
Couvrette v. Wisnovsky | D. Or., Dec. 12, 2025; fee order Mar. 23, 2026 | 15 nonexistent cases and 8 fabricated quotations across three summary judgment briefs | Claims dismissed with prejudice; $15,500 penalty; $94,704.38 in fees split 85/15 between lead and local counsel |
Flycatcher Corp. v. Affable Avenue LLC | S.D.N.Y., Feb. 5, 2026; fee award Oct. 9, 2026 | Fabricated citations; counsel blamed vLex and a cite-checking tool; explanation found not credible | Default judgment against the client; $38,877.59 in fees against counsel |
Withers v. City of Aberdeen | N.D. Miss., June 8, 2026 | Both sides' counsel filed briefs with hallucinated citations; local counsel did not check | Pro hac vice revoked and two-year bar for two lawyers; local counsel disqualified; $8,000 in fines |
Lexos Media IP, LLC v. Overstock.com, Inc. (patent) | D. Kan., Feb. 2, 2026 | Eleven misstatements of authority, including a nonexistent case, in a brief opposing exclusion of a technical expert; ChatGPT used | Five lawyers admonished; $12,000 in fines; one pro hac vice admission revoked |
In re Mitchell (USPTO OED) | Order of July 27, 2026 | AI-generated citations to a patent's specification and file history in a joint claim construction chart (E.D. Pa.) | Public reprimand |
Lnu v. Blanche | 9th Cir., June 3, 2026 | AI-related errors followed by a "repeated failure of candor" | $2,500 each; six-month suspension from Ninth Circuit practice; firm-wide AI disclosure for two years; bar referral |
Lacey v. State Farm General Insurance Co. | C.D. Cal., May 2025 | About 9 of 27 citations defective; CoCounsel, Westlaw Precision, and Gemini named in counsel's declaration | $31,100 against two firms; supplemental briefs struck |
Lifetime Well LLC v. IBSpot.com Inc. | E.D. Pa., Jan. 26, 2026 | False citations in a brief researched by a clerk using Lexis+ AI and Protégé | $4,000 personal sanction, paid to a legal aid project |
Cartagena v. Dixon | S.D.N.Y., Mar. 10, 2026 | Lawyer blamed Protégé; LexisNexis told the court he was never a subscriber | Public admonition |
Johnson v. Dunn | N.D. Ala., July 23, 2025 | Butler Snow lawyers filed motions with ChatGPT-fabricated citations | Public reprimand; disqualification; bar referral |
McCormick v. Texakoma Financial | E.D. Tex., June 11, 2026 | Fictitious case and fabricated quotations in a brief | Reprimand; $5,000; review of all 2026 filings; one-year verification certificates |
Gauthier v. Goodyear Tire & Rubber Co. | E.D. Tex., Nov. 25, 2024 | Two nonexistent cases drafted with Claude; a Lexis AI check did not flag them | $2,000; generative AI training; copy of order to the client |
Noland v. Land of the Free, L.P. | Cal. Ct. App., Sept. 12, 2025 | 21 of 23 case quotations in the opening brief fabricated | $10,000; State Bar referral |
Hill v. Foundation Media LLC | S.D.N.Y., Sept. 29, 2026 | Unverified quotations and a citation; firm later disclosed it had used Harvey to help prepare the filing | No sanction; the court called it "a wakeup call" |
Mata v. Avianca, Inc. | S.D.N.Y., June 22, 2023 | ChatGPT-invented cases | $5,000; letters to the client and to the judges falsely named as authors |
C. Five patterns in the decisions
First, legal-specific tools are in the record. CoCounsel, Westlaw Precision, Lexis+ AI, Protégé, vLex, Paxton, and Harvey all appear in sanctions proceedings or in counsel's explanations to the court, and roughly 30 entries in Charlotin's database mention a Thomson Reuters or LexisNexis product, many of them flagged as disputed by the vendor. In United States v. Farris, the Sixth Circuit observed that the principal brief's file name was "CoCounsel Skill Results" and held that the lawyer's duty to verify "remains true even when new tools are sponsored by trusted legal technology providers" (6th Cir.). Grounding in a licensed database reduces fabricated cases. It does not eliminate misquoted or misapplied real ones, and courts sanction the failure to verify either way.
Second, blaming the tool does not work. In Cartagena v. Dixon, a lawyer who blamed "the AI legal research platform Protégé in LexisNexis" was contradicted by LexisNexis itself, which wrote the court that he "was never an authorized user or subscriber to either the 'Lexis +AI' or 'Protege' services" (S.D.N.Y. opinion; LexisNexis letter). In Flycatcher Corp. v. Affable Avenue, the court found counsel's attempt to blame a cite-checking tool not credible. The Ninth Circuit's suspension order in Lnu v. Blanche was driven by what it called a "repeated failure of candor," and it stated the governing principle as precisely as any court has: "the rules are not violated at the point of research and drafting, but at the point of signing and filing" (9th Cir.).
Third, the sanctions are escalating in kind, not just in amount. In 2023, Mata v. Avianca produced a $5,000 penalty. Since then courts have dismissed claims with prejudice (Couvrette), entered default judgment against the client (Flycatcher), revoked pro hac vice admissions and barred lawyers from a district for two years (Withers), suspended lawyers from appellate practice (Lnu), and imposed $15,000 punitive sanctions per attorney plus the other side's appellate fees and double costs in a case involving fake citations (Whiting v. City of Athens, 6th Cir. 2026, where counsel refused to say whether AI was used). Large firms have paid in other currencies: Gordon Rees voluntarily repaid more than $55,000 in fees in a bankruptcy matter (Bloomberg Law), Goldberg Segalla and one of its lawyers were ordered to pay $59,500 in Cook County (Chicago Sun-Times), and a Morgan & Morgan partner sanctioned in Wyoming in 2025 was denied permission to appear in a Massachusetts matter in 2026 on the strength of that sanction (Law.com). Sanctions now follow lawyers across jurisdictions.
Fourth, policies are not controls. Sullivan & Cromwell's April 2026 letter to Chief Judge Martin Glenn acknowledged that "The Firm's policies on the use of AI were not followed in connection with the preparation of the Motion," and that a review process had not caught the errors, which opposing counsel did (Business Insider). Butler Snow had a policy too, and three of its lawyers were reprimanded, disqualified, and referred to the bar in Johnson v. Dunn (Reuters). In Withers, local counsel signed briefs they had not checked and were disqualified along with the lawyers who drafted them.
Fifth, the client often pays first. In Couvrette, a client's claims were dismissed with prejudice. In Flycatcher, a client lost by default. Malpractice exposure follows, and the insurance may not.
D. Patent practice and Texas courts are squarely in it
The patent bar has its own entries now. In Lexos Media IP, LLC v. Overstock.com, Inc., a patent infringement case, Judge Julie Robinson of the District of Kansas found "eleven misstatements of authority" in a brief opposing the exclusion of the plaintiff's technical expert, including a case, Hockett v. City of Topeka, of which the court wrote: "Neither this case nor the quotation exists." The court publicly admonished all five signing lawyers, fined four of them a total of $12,000, and revoked one lawyer's pro hac vice admission (D. Kan.). The USPTO's July 2026 reprimand in In re Mitchell, discussed in Part V, extends the problem from case law to the intrinsic record.
Texas courts have been early and specific. The Eastern District of Texas's Local Rule AT-3(m) cautions that some technologies "may produce factually or legally inaccurate content" and requires lawyers who use them to "review and verify any computer-generated content." In Gauthier v. Goodyear Tire & Rubber Co., Judge Marcia Crone sanctioned a lawyer $2,000, ordered generative AI training, and required him to give his client a copy of the order after a brief drafted with Claude cited two nonexistent cases that a Lexis AI checking feature had failed to flag (E.D. Tex.). In June 2026, in McCormick v. Texakoma Financial, Judge Amos Mazzant publicly reprimanded a lawyer, imposed a $5,000 penalty, required her to review every filing she had signed in 2026 and certify the result, and required a verification certificate on her filings in that court for a year (E.D. Tex.). The Tyler Division's April 2025 standing order requires a "Certificate of Generative Artificial Intelligence Usage" with every filing (E.D. Tex.), and the Beaumont Division's July 2026 standing order states that "The fact that AI was used to create the offending document will not excuse or absolve ethical infractions" (E.D. Tex.). The Fifth Circuit declined in 2024 to adopt an AI certification rule, reasoning that existing rules suffice, and then demonstrated what existing rules can do in Fletcher v. Experian, sanctioning a lawyer who "misled, evaded, and violated her duties as an officer of this court" (5th Cir.).
E. Judges are in the database too
The 32 judicial entries are a reminder that the problem is not confined to the bar. In 2025, two federal judges withdrew orders containing errors traced to a clerk's use of Perplexity and an intern's use of ChatGPT (FedScoop). In September 2026, it emerged that an Oklahoma state judge had admitted to investigators that a custody order citing at least two nonexistent cases was drafted with ChatGPT (Reuters). In May 2026, a California appellate court reversed a trial court that had relied on a fabricated case that opposing counsel had already flagged (LawLink). A survey of federal judges published in March 2026 found that 61.6% of respondents had used at least one AI tool and 45.5% had received no training; one judge called the "consistent reports of zombie cases and other instances where AI conjures law or facts" "terrifying" (LawSites).
F. Courts abroad are seeing the same thing
In Ayinde v. London Borough of Haringey and Al-Haroun v. Qatar National Bank, England's Divisional Court held that tools such as ChatGPT "are not capable of conducting reliable legal research," noted that 18 of the 45 authorities cited in one case did not exist, and referred lawyers to their regulators (Judiciary of England and Wales). The Solicitors Regulation Authority received 42 reports of AI-related wrongdoing between July 2025 and July 2026 (Law Gazette), and in September 2026 the Solicitors Disciplinary Tribunal struck off a registered foreign lawyer in its first AI case (Legal Futures). In Ontario, a lawyer who first blamed her staff and later admitted using ChatGPT received a six-month suspension (Law Times). In Victoria, a solicitor lost the right to practice as a principal (Victorian Legal Services Board).
The problem is not even confined to lawyers. Deloitte agreed to refund part of an A$440,000 Australian government contract after a report it delivered was found to contain AI-generated errors, including a misquoted Federal Court judge (AP). And in April 2026, South Africa withdrew its draft national AI policy over fictitious sources, with the minister explaining that "The most plausible explanation is that AI-generated citations were included without proper verification" (SAnews). A national policy on artificial intelligence, undone by artificial intelligence, is as clean an illustration as anyone could ask for.
G. The costs are spreading beyond sanctions
The sanction is often the smallest cost. In EPIC's 2026 survey of lawyers' professional liability insurers, seven of thirteen carriers reported an increase in AI-related claims over the prior year (Insurance Business). Carriers interviewed in August 2026 described AI as part of the fact pattern rather than a wave of claims, and noted that malpractice policies typically do not cover court-imposed sanctions, disciplinary penalties, or fines (The Daily Record).
Legislatures and courts are writing rules. California's SB 574, signed on September 30, 2026, adds Business and Professions Code section 6068.1 and requires attorneys to verify "the accuracy of all case and statutory citations" in AI-assisted work (Harris Beach Murtha). On September 18, the Tenth Circuit proposed a rule requiring a certification that a human "has independently verified the existence of all cited authorities" (Reuters). The federal rules committee paused proposed Federal Rule of Evidence 707 on machine-generated evidence in May to consider significant revisions (Redgrave).
Clients are moving too. Meta's legal operations leadership has reportedly told outside firms it will not pay for work AI could have done (Law.com), while an Association of Corporate Counsel survey found that nearly 60% of in-house lawyers see no noticeable savings yet from their outside counsel's use of AI and 59% do not know whether their firms use generative AI on their matters (ACC). In the United Kingdom, 81% of firms say they disclose AI use to clients, but only 7% of clients recall it happening (Clio).
Two newer risks deserve mention because they do not show up in any hallucination count. In United States v. Heppner, Judge Rakoff held that a defendant's reports generated with a consumer AI tool, using facts learned from counsel, were neither privileged nor work product, in part because the provider's terms allowed disclosure and training (Proskauer). And courts have begun to see prompt injection in filings: in August 2026, in Elliott v. New York Bariatric Group, a Connecticut trial court revoked a self-represented litigant's e-filing privileges after he hid white-text instructions aimed at any AI that might review his papers. Morgan Lewis has warned that "It should not be assumed that any current AI product can detect or defeat prompt injection" (Morgan Lewis). As firms connect agents to their document systems, every incoming document becomes a possible instruction.
Bottom line: courts treat a fabricated or misrepresented citation as the signing lawyer's failure to verify, whatever tool produced it. Using a legal-specific product does not change that analysis, blaming one makes it worse, and the consequences now reach clients, careers, and coverage.
VIII. Lawyers Using AI, Measured
The question that matters most to a managing partner is not whether a model hallucinates in a lab. It is whether a lawyer using the tool produces better work than a lawyer without it. The randomized evidence now answers that question consistently: faster, more polished, not measurably more accurate, with a hidden cost to skill formation.
A. The randomized trials
Schwarcz, Manning, Prescott, Barry, Cleveland, and Rich assigned 137 upper-level law students to six realistic legal tasks using Vincent AI, a retrieval-based legal tool; OpenAI's o1-preview reasoning model; or no AI. Productivity rose by 50% to 130% on five of the six tasks, and the authors report that both tools improved overall work quality. But in the pooled results the authors found no evidence that either tool improved accuracy. Across 768 completed tasks, graders found 18 hallucinated citations: 11 in the o1-preview group, 3 in the Vincent group, and 4 in the group with no AI at all (Schwarcz et al., Journal of Law and Empirical Analysis). Retrieval grounding did roughly what the vendors claim on fabrication. It did nothing measurable for legal accuracy.
Earlier work from the same research group points the same way. In one experiment, GPT-4 "only slightly and inconsistently improved the quality of participants' legal analysis but induced large and consistent increases in speed" (Choi, Monahan & Schwarcz). In another, "students at the bottom of the class saw huge performance gains with AI assistance, while students at the top of the class saw performance declines" (Choi & Schwarcz). The tool lifts the floor and can lower the ceiling.
The study closest to patent practice is a preregistered field experiment by Autor, Rodchenko, Martin, Iscenko, Strand, Pearl, and Ferere, released as NBER Working Paper 35720 in September 2026 and revised in October. Two disclosures matter: the experiment was funded by Google, and five of the seven authors disclose Google employment. The researchers gave 133 practicing patent lawyers at eleven U.S. intellectual property firms three months of access to an unreleased Google Labs patent-drafting assistant and had blinded expert patent attorneys score their work. AI-assisted drafts scored 0.34 standard deviations higher at 10 days and 0.38 higher at 90 days. Then the researchers removed the tool and asked everyone to redline a patent draft unassisted. The treatment group scored 0.32 standard deviations higher overall, but the gain was concentrated among lawyers with seven or more years of experience (0.45). Junior lawyers showed no average gain (minus 0.03, not statistically significant), and the revised paper describes their scores as bifurcated, "suggesting that AI assistance served as a springboard for some juniors and a cushion for others." The September version put the point more bluntly: "The largest gains from AI thus accrued to the lawyers who retained the least." Measured time savings on the drafting task were about ten minutes against a baseline of roughly 110 minutes (Autor et al., NBER Working Paper 35720; September version).
Vals had measured raw AI output at "six times faster than the lawyers at the lowest end, and 80 times faster at the highest end" (Vals). The gap between 80 times and 9% is the time a competent lawyer spends checking and integrating the output. That time is not overhead. It is the product. Firms that budget for the 80x and not the 9% will be surprised, and so will their clients.
The third study is not legal, but it names the hazard. Dell'Acqua and colleagues gave 758 Boston Consulting Group consultants GPT-4 and found that on tasks inside the model's capability frontier they completed 12.2% more tasks, 25.1% faster, at higher quality; on a task designed to sit just outside that frontier, they were 19 percentage points less likely to reach the correct answer than consultants without AI (Dell'Acqua et al.). The frontier is jagged and invisible from the inside. A pinpoint citation, a false premise, a fact buried in the record, and a recital that concedes your client's position all sit just outside it, and none of them announces itself.
B. What practitioners report
The surveys describe a profession using tools it does not trust. LexisNexis's 2026 survey of 543 UK legal professionals found 94% using AI for legal work, up from 61% a year earlier, 41% the year before that, and 11% in 2023, while the share worried about reliance on inaccurate or fabricated output rose to 83% from 57% in early 2024. Ten percent described themselves as "dependent on AI," and 77% believed AI was causing clients or the public to misinterpret the law (LexisNexis; Legal Futures).
Morae's July 2026 survey of 850 senior legal professionals found that only 33% trust the results of AI-assisted legal work, that 48% say people "always or often materially change AI outputs before they can be used," and that 67% are "concerned that the cost of human verification and oversight might outweigh the efficiency benefits provided by AI" (Morae). A Paragon Legal survey of 252 legal professionals found that 67% had overridden or corrected AI-generated legal output, 58% would not be comfortable submitting an AI-drafted document to a regulator or court, and 42% had little or no trust in the technology (Legal Cheek). The 8am 2026 Legal Industry Report, drawing on more than 1,300 legal professionals, found 69% using general-purpose AI at work while only 9% reported a written and actively enforced AI policy (LawSites). Thomson Reuters' Future of Professionals Report 2026, which spans law, tax, audit, and compliance, found that 34% of professionals use AI tools their organization has not sanctioned, and that AI met or exceeded expectations for 66% of professionals at organizations with a named AI strategy against 22% where there was none (Thomson Reuters). And in a 2026 report from Everlaw, ACEDS, and ILTA, only 41% of respondents named output verification as a needed skill (Above the Law). That last figure may be the most worrying number in this article.
Bottom line: the tools make lawyers faster and drafts smoother. They do not make finished work more accurate, and they may make the next generation of lawyers worse at the one skill the tools most require.
IX. The Strongest Case for the Vendors, and Why It Does Not Change the Answer
The vendors and their defenders have a serious case, and it deserves to be stated at full strength.
First, humans make mistakes too. In Vals' October 2025 research study, the lawyer baseline scored 71% on accuracy, below every AI tool tested. In Legal Benchmarks' drafting study, human lawyers were reliable on 56.7% of tasks, slightly below the AI tools, and they raised no explicit risk warnings at all. Any honest comparison should be to the lawyer the client would otherwise get, not to perfection.
Second, the products change monthly. A 2024 benchmark measures a product that no longer exists, and even a 2026 benchmark will be stale by spring.
Third, zero is the wrong standard. As Harvey's cofounder put it, "One common misconception in legal is that an AI system needs to be 0 hallucinations to be useful. Our experience has been that these systems are incredibly valuable and the rate of hallucinations will continue to decrease" (LawSites).
Fourth, the court record is skewed. More than half the entries involve self-represented litigants using consumer chatbots, most entries do not identify the tool, and the lawyers involved are disproportionately solo and small-firm practitioners without enterprise tools or review layers.
Fifth, the strict benchmarks are harsher than practice. Lawyers do not ask an agent to complete a twenty-step assignment unsupervised, and a partner who sends a draft back for one missed point does not conclude the associate is useless.
Each point is correct as far as it goes. None of them changes the answer, for four reasons.
The first is detectability. The relevant comparison is not error rate alone but error profile. A human associate's mistake usually has a traceable reason and looks like a mistake. An AI's wrong pinpoint, invented quotation, or unsupported date is formatted exactly like a right one, delivered with the same fluency, and placed exactly where reviewers skim. The research in Part V shows that the errors that remain are the ones that are hardest for both machines and tired humans to catch.
The second is accountability. The vendor's terms disclaim responsibility. The lawyer's signature does not. The Ninth Circuit's formulation, that the rules are violated "at the point of signing and filing," means that every error the tool makes is legally the lawyer's error, and every error the lawyer misses is a candor problem waiting to happen.
The third is cost. Zero hallucinations is indeed the wrong standard. The right standard is the total cost of producing correct work, including verification. On the measured evidence, that cost has not fallen nearly as far as the demos suggest, and two-thirds of senior legal professionals in the Morae survey worry that verification may cost more than the efficiency it buys.
The fourth is that the vendors' strongest argument cuts against their behavior. If products improve monthly, the answer is to test them monthly, in public, on frozen question sets. The vendors that have declined to be tested cannot also claim that the only available independent numbers are out of date.
Bottom line: the vendors are right that their tools are valuable and wrong that their value can be assessed without independent measurement.
X. Why the Market Has Not Produced the Number
It is worth asking why, two and a half years on, the most basic fact about the most consequential tools in legal practice is still unknown. The answer is incentives, and understanding them tells a buyer what to do.
No vendor gains from a public number unless it wins. Independent benchmarks depend on vendor opt-in, so the vendors with the most to lose can decline, and have. Buyers rarely insist, because procurement is driven, as Guo's interviews found, by security, integrations, and existing contracts, and because the people who sign the contracts are seldom the people who read the cited pages. Products change monthly, which gives every vendor an evergreen reason to dismiss any test as stale. And the measurement itself is hard: the same outputs can score 15% or 1% depending on the reference database, a government-built answer key can turn out to be incomplete, AI checkers disagree with each other by more than an order of magnitude, and "hallucination" can mean two different things.
The vendors are not wrong that a fair test is difficult. They are wrong that it is impossible, because Stanford, Vals, Linklaters, Legal Benchmarks, the DELTA project, and Artificial Analysis have each run one.
A credible test is not mysterious. It uses a frozen, public question set with a held-out portion. It reports fabrication, misgrounding, wrong pinpoints, contradictions of the record, omissions, and refusals as separate rates rather than one blended "accuracy." It names the reference database, the product version, and the base model. It uses blinded human graders with a published adjudication log, and reports how often the graders agreed. It compares the product to its own base model on identical prompts. It reports cost and time, including verification time. And it is rerun on every major release. Until a vendor submits to that, the appropriate posture toward its accuracy claim is the posture a court takes toward an unverified citation.
The Layer-by-Layer Answer
Layer | Has accuracy improved since 2024? | Strongest evidence | What remains |
Frontier models | Yes, sharply on fabrication | Fictitious authority from about a third of answers to 9% (Linklaters); major-error responses 22.0% to 4.8% (OpenAI, self-reported); LegalBench 77.7% to 88.56% | Deference to users; wrong pinpoints; regressions between releases |
Research platforms (Westlaw, Lexis) | Probably; never independently measured | No rerun of the Stanford protocol; 2026 statutory-survey test at 58% and 64% | Vendor opt-outs from public benchmarks; unpublished definitions of accuracy |
Workflow wrappers (Harvey and peers) | On content and workflow, yes; on reasoning, no | Harvey top AI score on 5 of 6 tasks (2025); legal vs. general tools 57.6% vs. 58.3%; Thomson 0.83 vs. GPT 5.4 0.82 with the same content | One independent benchmark of Harvey, in 2025; uneven real-world use |
Long agentic tasks | No | About 94% of criteria but 20% to 25% of tasks (LAB); 9.4% with a hallucination gate (LAB-AA); 55.29% (Vals research); 15.0% (DELTA) | The all-pass wall; verification load grows with output length |
Lawyers using AI | Faster and more polished; not more accurate | 50% to 130% productivity, no pooled accuracy gain (Schwarcz); juniors show no retained gain (Autor) | Skill formation; the 80x versus 9% gap |
Courts | Worse | About 120 to 2,184 decisions; 2.7x year over year; dismissals, default judgments, suspensions | Detection, not error rate; costs spreading to clients, insurers, and admissions |
Short Answers to Common Questions
Is Westlaw's AI still hallucinating 33% of the time? Nobody outside Thomson Reuters knows. The 33% was the 2024 result for Westlaw AI-Assisted Research under Stanford's broad definition. Thomson Reuters has since rebuilt its products around CoCounsel and its own model, claims approximately 90% accuracy from internal testing, and has not submitted to an independent retest.
Is Lexis+ AI more accurate than Westlaw's AI? In the 2024 Stanford study, yes: about 17% against 33%. A separate 2025 test put Lexis+ AI at 20% hallucinated on a different question set. The 2026 statutory-survey test found the two platforms fail differently, with Lexis finding only 29% of the relevant provisions. None of this measures the current conversational research products.
Does using a legal-specific tool instead of ChatGPT protect me from sanctions? No. CoCounsel, Westlaw Precision, Lexis+ AI, Protégé, vLex, Paxton, and Harvey have all appeared in court records involving defective filings. Grounding reduces fabricated cases. Courts sanction the failure to verify.
Should I use an AI cite-checker? Yes, as a first filter. The 2026 studies show automated checkers catch substituted or fabricated cases well and miss a large share of wrong pinpoints. The checker is a floor, not a signature, and it is never a substitute for reading the source.
Can Harvey be trusted for legal research? The only independent benchmark that scored Harvey tested document tasks in February 2025, not research. Harvey's own help center flags exhaustive research as a use that "may produce less reliable results," and its own data shows its research advantage comes from connectors and licensed content rather than reasoning. On my own team, lawyers have seen it return erroneous information and hallucinated authority regularly, even when grounded in LexisNexis content. Verify its citations and its legal framing the way you would any other tool's.
Which model is most accurate right now? It depends on the task and the week. On Vals' Legal Research Bench, three models are tied at 55.29% fully correct. On the hallucination-gated version of Harvey's agent benchmark, one model averages 0.03 material hallucinations per task but still passes fewer than one task in ten. The leaderboards change monthly, which is the best argument for testing on your own matters.
Are the percentages going down? For fabricated authority in raw models, yes, sharply. For the research platforms, probably, but unmeasured. For long tasks graded strictly, the best models still fail most of them. For lawyers' finished work, accuracy is flat. For the court record, the trend is up.
XI. Recommendations
The tools are good enough to use and not good enough to trust. The evidence says trust has to be built into the workflow rather than purchased with the license. What follows is what I would do, and in several cases what my own firm does.
For lawyers who sign filings and drafts
Read the cited page, not the citation. The live risk in 2026 is a real case that does not say what the brief says, or says it on a different page. Existence checks and citators do not catch that. A human reading the pinpoint does.
Verify against an independent source, never with a second model alone. In In re Mitchell, a second AI tool "verified" citations to a file history that did not contain them. A different model is not a different source. Pull the opinion, the specification, the transcript, or the exhibit.
Treat facts about your own record as citations. Dates, amounts, parties, deposition pages, column-and-line cites, and prosecution history statements are where the 2026 hallucinations live. Check them against the record with the same rigor you apply to case law.
Read the recitals, definitions, and boilerplate as carefully as the operative terms. AI drafts pad agreements with standard-form language, and padding is where the tool's misunderstanding of your objective hides. In my standstill test, the dangerous sentence was a recital, which is exactly where a tired reviewer skims.
Assume every long AI work product contains at least one error until someone has ruled it out. The all-pass benchmarks say that assumption is correct more often than not, even for the best models.
Never give the tool your conclusion and ask it to confirm. Deference to a confident user is the failure mode that has improved least since 2024. Ask open questions, then ask for the strongest contrary authority.
Cite-check the final draft, not an earlier one. Citations migrate and mutate through revisions. The cite-check that counts is the one performed on the document that gets filed.
Use automated checkers as a floor and your own reading as the signature. Automated detection catches fabricated and substituted cases well and misses wrong pinpoints badly, while generating false alarms of its own.
Read the standing order of every judge you appear before, and certify accurately. In the Eastern District of Texas alone, the local rules, the Tyler Division, and the Beaumont Division each address AI, and certification language varies by court and by judge.
If an error gets through, own it immediately and completely. Courts have been harshest on lawyers who minimized, blamed the tool, or misstated how they used it. In Cartagena, the vendor itself supplied the evidence that the lawyer's explanation was false.
For patent litigators and prosecutors
Build intrinsic-record verification into every claim chart, claim construction brief, contention, and expert report. Check every column-and-line cite, figure reference, and file-history quotation against the PDF before anything is served or filed. A 200-row chart is exactly the long work product the strict benchmarks say will contain errors.
Remember the USPTO's standard. "Simply relying on the accuracy of an AI tool is not a reasonable inquiry." And the Office of Enrollment and Discipline has now disciplined a practitioner for AI errors made in district court litigation, beyond the filings the Office itself receives.
Keep patentability and invalidity judgments with the experts. Benchmarks of model performance on novelty and nonobviousness remain only modestly better than chance. Use the tools to find and organize prior art, and leave the judgment about what it means to someone who can defend it under oath.
For firm leadership and procurement
Buy the layers separately in your own mind, even when they come bundled. A legal AI subscription combines model access, licensed content, workflow, and verification tooling. Price each, test each, and ask what the product does that the model underneath it does not.
Make independent testing a condition of procurement. Ask every vendor which public benchmarks it has entered, declined, or withdrawn from, and why. Ask for results reported by error type, with the reference database, the checker, the product version, and the base model named. A vendor that cannot answer is telling you something.
Run your own frozen test set. Twenty to fifty matters from your practice, with answer keys built by your partners, rerun on every major product release and on the base model alone. Include at least one task like my standstill test, where the failure mode is judgment rather than citation. The cost is a few partner-days. The alternative is discovering the answer in a show-cause order.
Budget verification as a named line of work, not as overhead. The measured time savings for practicing lawyers is on the order of 9%, not 80 times, and the difference is review. Under ABA Formal Opinion 512, lawyers who bill hourly must bill their actual time and may not bill clients for learning a tool (ABA). Firms that staff and price as though review time does not exist will either under-review or under-recover.
Write the AI policy down, train to it, and audit compliance. Sullivan & Cromwell and Butler Snow both had policies. Sample filed documents for compliance the way you sample time entries.
Require two-layer review for AI-assisted filings, and make the second layer a human reading sources. The second reviewer in Sullivan & Cromwell's matter and local counsel in Withers both missed errors that were caught as soon as someone outside the drafting team read the sources. Local counsel's signature has to mean something.
Revalidate on every model change, and require vendors to tell you when the model changes. The o3 regression, the 62%-to-97% citation swing in Harvey's own data, and Harvey's move to a post-trained open-weight model all show that a workflow validated on one model is not validated on the next.
Know which model and which infrastructure touched client data. Get contractual commitments on training, retention, sub-processors, and model provenance. Heppner shows what is at stake when AI work product sits on terms that allow disclosure.
Treat incoming documents as untrusted input for agentic tools. Opposing filings, data-room documents, and emails can carry hidden instructions. Do not give an agent the power to send, file, or delete on the strength of untrusted documents without a human approving the action.
Review your professional liability coverage with AI in mind. Insurers are beginning to see AI-related claims, and standard policies typically do not cover sanctions or fines. Have the answer before the application asks.
Measure outcomes, not licenses. Track errors caught in review, verification time, rework, and client write-offs by tool and by task. License utilization is a vanity metric. Error rates on your own matters are the number that matters.
For training and talent
Protect the skill that makes the tools safe. The patent-drafting trial found that junior lawyers using AI for three months showed no average gain on unassisted work, and a federal judge has now warned in a written order that over-reliance on AI risks stunting their training. Schedule AI-off work for juniors, make unassisted cite-checking a required rotation, and evaluate associates on their ability to find the error in an AI draft, not just to produce one.
Teach the error taxonomy explicitly. Associates should be able to distinguish fabrication, misgrounding, wrong pinpoint, contradiction of the record, omission, and the judgment error that cites nothing at all, and they should know which tools catch which. In one 2026 survey, only 41% of respondents named output verification as a needed skill.
Pair the tool with the expert, not the novice, on anything that will be filed. The randomized evidence says the tool amplifies judgment that already exists and substitutes poorly for judgment that does not.
For in-house counsel and clients
Put verification requirements, not just disclosure requirements, in outside counsel guidelines. Require that every citation in a filed document was read by a licensed lawyer, and reserve the right to audit. Refusing to pay for AI-replaceable work addresses cost. It does not address risk.
Ask your firms which AI products they use, how they validated them, and when they last revalidated them. If the firm cannot answer the third question, the first two do not matter. Then ask whether the time savings appear on your bill.
For vendors, courts, and regulators
Vendors: publish the rubric or stop publishing the number. Enter independent benchmarks or explain publicly why not, report errors by type, and publish the comparison against your own base model. The 2024 numbers can be replaced any time a vendor agrees to be tested.
Courts and regulators: standardize, publish, and fund. Uniform certification language, a public registry of standing orders, consistent treatment of candor failures, and free public access to primary law for self-represented litigants, who account for more than half the database, would do more for the error rate than larger fines.
On AI in general
Treat reliability as a property of a workflow, not of a product. The same model scores 94% per criterion and 25% per task. The same outputs score 15% or 1% depending on the yardstick. The same lawyer is faster with the tool and no more accurate. What changes the outcome is the structure around the tool: the quality of the question, the length of the deliverable, the reference the output is checked against, and whether a competent person reads the result.
Put the human where the human adds information, not where the human adds a signature. Rubber-stamp review is the failure mode in nearly every sanctions case in this article. Verification is skilled work, and it should be staffed, trained, and paid that way.
Expect the ground to keep moving, and build the capacity to test rather than the habit of buying. The labs now sell legal products, the content owners now build models, and the wrappers now post-train their own. Whatever you validated this year will need validating again next year. The firms that win the next five years will not be the ones with the most licenses. They will be the ones that know, on their own matters, exactly how often their tools are wrong.
Comments