How to Test Legal AI on a Matter Your Firm Already Knows

The easiest legal AI demonstration to pass is one where the buyer does not know the answer.

A polished result can look impressive when nobody in the room knows which documents matter, which facts are missing or how long a proper review should take. A closed matter gives the firm something much more useful: a reference point.

The team already knows what happened. It knows the difficult documents, the unresolved conflicts and the mistakes a system is likely to make. That makes the matter suitable for a controlled test.

Choose the matter before choosing the winner

Use a closed, de-identified or internally approved matter that is representative of the work the firm wants the product to perform.

The matter should be large enough to expose cross-document problems, but not so large that the firm cannot build a credible reference record. It should contain at least one factual conflict, one important item that is easy to overlook, and a question the team can answer from its own prior work.

Avoid a matter supplied by the vendor. A vendor-selected example may be useful for learning the interface, but it does not give the buyer an independent basis for assessing the result.

The test should also be task-specific. "Understand the matter" is too broad. A stronger instruction might be:

• build the material factual record relevant to three defined issues;
• identify the evidence supporting and contradicting a pleaded allegation;
• prepare a source-linked chronology for a specified period;
• identify expected documents or events that are missing from the supplied record.

Build the reference record first

The firm's reference record should be completed before the products are run.

It does not need to contain every fact in the matter. It needs a defensible list of the facts that are material to the chosen task, the sources supporting them, and the known conflicts or gaps.

Record the basis for materiality. A fact may be material because it changes an element of a claim, affects a limitation issue, contradicts a witness, alters the damages position or points to further investigation. Writing that down prevents the team from changing the answer key after seeing which product performed best.

The reference record is still a human work product. It may contain mistakes or omit something the product finds. If that happens, the team should review the proposed addition and update the reference record where justified. The goal is a fair evaluation, not defending the first human answer at all costs.

Fix the inputs and instructions

Give each product the same documents, the same task and the same amount of contextual information.

Record:

• the number of files and pages supplied;
• any files that failed or were excluded;
• the exact instruction;
• the settings, model and product version where visible;
• whether the product received the issues, pleadings or other matter context;
• the time spent configuring the test.

A product that requires different inputs may still be valuable. The difference should be part of the result rather than hidden in the setup.

Run the initial test without revealing the private reference record. Known contradictions and omissions should remain concealed until the output is assessed.

Measure the work that matters

A useful test should measure more than whether the final prose reads well:

• Matter coverage: did the system process every expected document and page, and were failed, unread or excluded files visible?
• Material-fact recall: what proportion of the material facts in the reference record did the product surface?
• Factual precision: of the factual propositions the product produced, how many were supported by the supplied material?
• Source fidelity: did the cited page support the precise proposition beside it, including the actor, date and surrounding context?
• Conflict and gap handling: did the product preserve inconsistent accounts and identify material that appeared to be missing?
• Correction persistence: after a lawyer corrected a fact, did later work use the corrected version?
• Time to usable work: how long did the team spend on setup, generation, source review, correction and omitted-material review before it was prepared to use the result?

Those measures should be kept separate. A product can have high precision and poor recall. It may avoid unsupported claims by returning very little. Another product may surface more of the matter while creating more review work. One headline percentage will obscure that difference.

Review the omissions deliberately

Most product demonstrations focus on what appeared in the answer. The known-matter test should also examine what did not.

Compare the product's factual record against the reference record and classify each missing item. Was the source unread? Was the fact buried in an attachment? Did the system find the document but fail to treat the passage as material? Did the instruction exclude it?

The remedy depends on the cause. A file-processing failure, an extraction failure and a materiality judgment are different product problems.

Ask the reviewers to inspect a sample of documents that the product did not use. A list of citations shows where an answer came from. It does not establish that the system considered the rest of the matter.

Correct the record and run a second task

The first run tests extraction and output. The second run tests whether the product can carry reviewed work forward.

Correct several known errors or classifications. Then ask for a different work product that depends on the same facts, such as a witness brief, deposition questions or a summary of the evidence on one issue.

Record whether the product:

• used the approved corrections;
• recreated an earlier error;
• disclosed that a change was pending;
• preserved the link to the original source;
• applied the correction only to one user or across the matter.

A system that requires the lawyer to restate every correction is providing generation, not persistent matter state.

Compare the products at the task level

Do not average unrelated activities into one vendor score.

A legal research tool should be tested on legal research. An eDiscovery platform should be tested on the relevant collection and review task. A factual-record product should be tested on whole-matter factual coverage and reviewability. A general drafting model should be tested on a controlled drafting task with defined inputs.

The same firm may choose different products for different jobs.

The Avoiding the Hype Trap article covers the broader vendor questions around security, support and implementation. The Legal Ops pilot guide covers ownership, users and rollout. The known-matter test supplies the evidentiary test inside that pilot.

Record the limitations

A closed matter is not a perfect prediction of future performance.

The product may behave differently on another practice area, language, document quality or volume. The firm's reference record may reflect the lawyers' own interpretation of materiality. A single test cannot establish a universal accuracy rate.

It can establish something narrower and more useful: how the product behaved on a real task where the firm could identify both the surfaced errors and the missing work.

NIST's work on AI testing and evaluation repeatedly stresses that measurement depends on the context in which a system is used. The same principle applies here. The firm should test the product it intends to buy, on the work it intends to give it, with the review process it will actually use.

Notes and sources

• NIST AI test, evaluation, validation and verification, on context-specific measurement and evaluation.
• NIST ARIA, which combines model testing, red-teaming and field testing rather than relying on one performance measure.
• NIST AI Technology Evaluation, an example of using blind data and a sequestered test environment.
• Verification in Legal AI Is a Design Problem, Mary's practical framework for checkable legal AI workflows.