Avoiding the Hype Trap in Legal Tech: What to Ask Your Vendor

A legal AI product can look convincing in a demonstration and still be a poor fit for the firm’s work.

The most useful due diligence starts with a matter and task the firm already understands. That gives the buyer something concrete to test and makes vague product claims easier to challenge.

What exact job is the product designed to perform?

Ask the vendor to define the task narrowly. “Litigation AI” can mean legal research, discovery review, factual extraction, drafting or matter administration. A product that performs one well may not be the right system for another.

The vendor should be able to explain the intended input, the output it produces and the review process expected of the lawyer.

What does the product actually read?

For matter work, confirm whether the system processes every uploaded page, how unreadable or failed files are reported, what happens when new material is added and whether the user can see which sources were considered.

A citation to one document does not establish coverage of the whole matter.

How does a lawyer verify the output?

Ask to see the review path in the product. Can the user move from a factual or legal claim to the exact passage relied on? Is the surrounding context visible? Can the user correct an error, and does that correction persist into later work?

Mary’s verification whitepaper sets out five controls that firms can ask a vendor to demonstrate.

What does the evaluation measure?

A headline accuracy number is difficult to interpret without the task, dataset, scoring method and error categories. Ask whether the evaluation counts unsupported claims, factual omissions and failures to process part of the record.

The Bluebook study discussed here is useful because the researchers fixed the task and answer key before comparing different system designs.

What happens to client data?

Review the service and contract the firm will actually use. Confirm processing location, retention, training use, access controls, deletion, encryption, incident response, subprocessors and the vendor’s security assurance.

What implementation work is required?

Ask who configures the product, trains users, supports the pilot and responds when the output is wrong. Find out whether it connects to the firm’s existing document and practice systems or creates another disconnected copy of the matter.

Run a controlled pilot

Use a closed matter and define success before the pilot begins. Record review time, source accuracy, material omissions, user adoption, support required and the time needed to reach a usable result.

How Legal Ops Teams Can Drive Tech Adoption gives a practical structure for running that test.

The best vendor answers are specific and demonstrable. The firm should be able to see what the product does, where it fails and how much work is required to use it safely.