Case study · Disputes · AustraliaHow Clayton Utz chose its legal AI.
The firm benchmarked five AI platforms against its own disputes lawyers on a live matter, then rolled out the one its associates trusted to be exhaustive.
- AI platforms tested
- 5
- live complex litigation matter
- 1
- lawyers set the human benchmark
- 20
- rollout after the pilot
- Full
Ex. 1The method
How they ran the test.
A fair test, set up before anyone saw a result.
A fair test, set up before anyone saw a result.
- 1Set a human benchmark20 of the firm’s lawyers reviewed a complex litigation matter end to end. Their work became the baseline every platform was judged against.
- 2Run the same matter through five platformsThe same documents went through five leading AI platforms, each set up for the same task.
- 3Score against the firm’s own workDisputes associates scored each platform’s output against the baseline, on weighted criteria the firm set in advance.
- 4Decide on the evidenceMary won the two criteria weighted highest. The pilot became a full rollout across the disputes practice.
Ex. 2The criteria
What decided the result.
| Criterion | Weight | What it tests | Result |
|---|---|---|---|
| Extraction completeness | Highest | Did the platform find everything the lawyers found, and what did it add? | Mary ranked first |
| Verification traceability | Highest | Can a lawyer check each fact at its source page, quickly? | Mary ranked first |
- Extraction completenessMary ranked firstWeight: HighestDid the platform find everything the lawyers found, and what did it add?
- Verification traceabilityMary ranked firstWeight: HighestCan a lawyer check each fact at its source page, quickly?
“We needed the tool that could find what others missed, and prove it. Mary was the only platform where our associates felt confident they’d been exhaustive.”
Ex. 3Your turn
Run the same test on your matter.
We’ll help you set it up, using your matter and your lawyers as the benchmark.
We’ll help you set it up, using your matter and your lawyers as the benchmark.
- 3.1
Pick a live matter you know well
The test only means something if your team already knows what’s in it.
- 3.2
Benchmark against your own lawyers
Your team’s review is the baseline. Every platform is scored against it.
- 3.3
Weight what matters
Completeness and traceability decide whether a lawyer can rely on the output.
Results relate to the tested matter and configuration.