Verification in Legal AI Is a Design Problem
Lawyers remain responsible. Legal AI should make its work easy to check before they sign.
When legal AI changes information you have already verified
A new study of frontier models shows how legal AI can alter correct source information after retrieval, and why the system around the model can materially change the result.
By Harry Raworth, Co-Founder, Mary Technology.
Matthew Dahl and Eric Martínez open their new working paper, Bye-bye, Bluebook? Automating Legal Drudgery With AI-Augmented Rule Following, with an incident that is more difficult to catch than the familiar example of an AI inventing a case.
In 2025, a lawyer at Latham & Watkins had already found the relevant source, checked it and supplied a chatbot with the correct publication title, year and link. She asked the model to format the citation. It changed the title and authors.
The authors describe the incident as different from a hallucination in the conventional sense. The model did not need to find a source or fill a gap in the research. The correct information had already been provided. The error entered when the model handled that information in the next step.
That creates an awkward form of legal AI verification. If a model invents an authority, checking whether the authority exists should expose the problem. Here, the authority is real. A source check can confirm that the document exists while missing the fact that the model changed information the lawyer had already verified.
What the Bluebook study actually tested
Dahl and Martínez use Bluebook citation formatting because it gives them an unusually clean way to measure this behaviour. The rules are public and formally specified, and there is an answer against which the output can be checked.
They assembled 2,058 Bluebook queries and tested six flagship models. The main open-formatting set contained 914 tasks where the model received truthful and complete information about the underlying authority and had to produce the correct citation.
Across those open tasks, the paper reports a weighted accuracy of 42.6% for producing a fully Bluebook-compliant citation.
That number needs some context. The scoring is deliberately strict and includes required typography, so it would be wrong to read 42.6% as meaning that every other response was completely unusable. The paper also records substantive failures, including incorrect parties, courts and dates, fabricated citation details, incorrect subsequent history and missing author information.
The model often had enough information to produce the right answer. The failure occurred while turning that information into the legal output.
The same model behaved very differently depending on the system around it
The researchers then tested several ways of improving performance with GPT-5.6 Sol.
- Standard model: 42.2%. Baseline.
- Reasoning mode: 47.5%. More test-time reasoning.
- Relevant rules supplied: 63.1%. The applicable Bluebook material was placed in context.
- Structured pipeline: ~79%. The model parsed the information; deterministic software applied the rules.
Supplying the rules helped substantially. It still left more than one-third of the open tasks failing.
The larger improvement came when the researchers changed how the work was divided. The language model classified the citation type and extracted the relevant information into structured fields. Deterministic software then applied the Bluebook rules to those fields.
Across all six models, the structured pipeline increased pooled accuracy from 42.6% to 75%, and the best end-to-end result reached 85.5%.
One comparison makes the effect especially clear. DeepSeek V4 Flash was the weakest model in the zero-shot test at 22.7%. Inside the structured pipeline it reached 77%, above the strongest model operating on its own.
The underlying model is therefore only part of the product story. What the model is asked to do, what is represented outside it and which steps are handled by ordinary software can materially change the result.
Two different places for an error to enter
The paper separates the failures into two categories.
An extraction failure occurs when the model incorrectly identifies or structures the information in the input.
A rule-following failure occurs when the model has the relevant information but applies the Bluebook rules incorrectly.
The structured system still relies on the model for the first job. Once the information has been extracted into the required fields, the rule application is handed to deterministic software.
That separation lets the researchers see where the remaining errors are coming from. When their deterministic formatter was given correctly structured inputs, it achieved 100% accuracy on the main structured test set and 98.2% on a separate held-out set. The errors in the full pipeline therefore came from the model-driven classification and parsing stage rather than the formatter applying the rules.
This is a useful way to think about legal AI system design. Language models are well suited to interpreting unstructured language. Some later steps can be represented explicitly, checked or executed consistently in software. Leaving those steps to free-form generation creates another opportunity for information to change.
A real source does not prove faithful use of the source
Legal AI products increasingly make source material visible. That is valuable because it gives the lawyer somewhere to check the work.
It does not, by itself, establish that the model used the source correctly.
The Latham example shows the difference. The source existed. The lawyer had already verified it. The model still changed the information while producing the next artefact.
This is why source provenance and output fidelity should be treated separately when firms evaluate legal AI reliability. A product can be grounded in the right document and still misstate what it takes from that document.
What carries over to litigation
The percentages in this paper belong to Bluebook citation formatting. Building the factual record of a litigation matter is a much less bounded problem.
Litigation files can contain inconsistent accounts, inferred dates, missing attachments, repeated names and factual significance that changes with the issues in dispute. There is no equivalent of a Bluebook answer key for the matter as a whole.
That makes the Bluebook study a relatively favourable test of precise legal execution, rather than a direct proxy for factual work in litigation. The researchers make the same limitation clear: citation formatting is unusually suited to their architecture because the rules are well specified and the inputs are comparatively easy to structure.
The concern still carries over. If a model can change correct information while performing a bounded task with an objective answer, firms should test what happens to source information as it moves through more complex legal workflows.
The paper itself identifies cross-document work and the extraction of key facts and chronologies as areas where transparent benchmarking remains limited. For litigation, that points toward a broader question than whether the final answer sounds right: how much of the factual record did the system capture, what evidence supports each factual claim, and what did it fail to capture?
Questions worth asking a legal AI vendor
A firm assessing a legal AI product can use the paper to ask more specific questions about how the product is built:
- After a source is retrieved, which steps are still performed by the language model?
- Which information is stored in structured form rather than recreated during each generation?
- Where are deterministic checks or ordinary software used instead of another model call?
- When a lawyer corrects a fact or source detail, does that correction persist?
- How has the product been tested for errors introduced after correct source information enters the system?
- Does the evaluation measure only the quality of surfaced answers, or also factual omissions?
These questions are more informative than the name of the underlying model or a single headline accuracy number.
The verification boundary moves downstream
The paper moves the reliability question one step further into the workflow.
In the Latham example, the substantive research and source verification were already complete when the error entered. The model received the right information and changed it while performing the next task.
A review process that stops at “is there a source?” tests only one part of that workflow.
