Who Decides What Counts as a Seven?

By Catherine Thompson, Head of Mary Legal, U.S., Mary Technology.

eDiscovery today lets the party holding the records decide what a request means, apply that reading in a black box, and then certify the result with statistics built on the same reading. That arrangement hardens information asymmetry instead of reducing it. It also loses sight of what discovery is for: letting the requesting party surface the facts it needs, including the facts it needs to win the argument about what else it is entitled to see.

This paper imagines a model of the future. Both sides can analyze records for facts, not just documents. And the producing party’s operative interpretation of a request is treated as what it is: a fact about whether a reasonable search was done, and therefore open to scrutiny.

“Go Fish” with House Rules: Two Problems with Modern Discovery

Despite Judge Peck’s vision in the seminal Technology-Assisted Review (“TAR”) case Moore v. Publicis Groupe (2012) (commonly called Da Silva Moore) that TAR would solve the problem, modern discovery still works a lot like a game of Go Fish.[1] The requesting party asks, in effect, “Do you have any sevens?” The responding party (or “holder”) looks at its own hand and decides what to give up.

This paper is about two house rules that make the game lopsided:

The holder effectively decides what counts as a seven. Though the requesting party defines a population of documents in words, it is up to the producing party to interpret those words and apply them out of view[2] to decide what is collected, searched, and produced, all while knowing far more about how rich its own record populations are.

The holder may leave a quarter of the sevens unfound. Courts have generally treated estimated recall of 70% to 85% as reasonable and proportionate.[3] Though on paper no one may withhold documents known[4] to be responsive, the holder can effectively shortcut a complete retrieval of the document collection. Because the holder also defined the collection, the quarter left unfound is a quarter of a set it drew itself. Still, the system offers no standardized way to scrutinize how the collection was defined.

Both rules shortchange the requesting party in a ludicrous game of circularity: It must guess what’s in the card deck (the “document population”) to reasonably tailor requests. To assert there may be “more” in the hand (the “document collection”), it effectively must prove by inference the deck is richer than the holder says. Still, the holder gets to choose what’s in the hand based on an asymmetrical understanding of richness from the start.

Even worse, the effects of both rules are compounded by statistics and legal doctrine.

“Validation” Launders the Interpretation; “Work Product” Shields It

TAR did not create this asymmetry: Manual review also left interpretation and search construction to the producing party. But modern claims of “validation” of such construction do add a statistical seal, and claims of work product protection add a shield.

As to validation’s seal: In a typical TAR protocol, the producing party’s reviewers code as “responsive” a seed set of documents a model learns from. The same reviewers, under the same reading of the requests, select and code the sample of the document collection used to estimate recall. If either reading is narrow, the model learns the narrow line and the samples confirm it. A production can thus accurately report “90% recall” with a mathematically correct “confidence interval,” where the only thing anyone is actually “confident” about is how well the model surfaced records the producing party coded as “responsive.”[5] Meanwhile, both the responsive definition and document collection searched may have been defined too tightly. Many protocols also narrow the collection with keyword searches before TAR begins. But unless recall is then measured against the full pre-cull collection, documents excluded by those searches fall outside the recall estimate entirely.

Similarly, even when reviewers’ “accuracy” in interpreting the requests to define the collection and score the output is “measured,” it is against a “gold standard” set not by “ground truth” but by senior attorneys—who frequently disagree with each other, and sometimes even with themselves.[6]

The result is outcome statistics standing in for scrutiny of how (1) the lines underpinning those statistics were originally drawn and (2) the right to draw those lines in the first place disproportionately burdens requesting parties at every step: An unexamined interpretation goes in. A “certified” number comes out.[7] Each interpretation not only remains concealed; it becomes harder to challenge, because a challenger must now argue against what looks like measurement.[8]

As to the doctrinal shield: Producing parties routinely argue, and some courts and commentators agree, that how a producing party interprets a request to select responsive documents over others is protected “work product” (i.e., counsel’s mental impressions about what the words mean).[9] But that treatment conflates two different things:

• The strategy behind an interpretation, including how counsel advised the client, is work product and should stay protected.[10]
• The line as applied, meaning which categories of records were treated as in or out of the request, is a fact, generally predicated on other relevant facts. That line, and the facts behind it, define whether the party conducted the “reasonable” search that Rule 26(g) requires counsel to certify.

Federal Rule 34(b)(2)(C)’s requirement that an objection state whether responsive materials are being withheld on the basis of that objection[11] illustrates a step toward a possible solution: Discovery rules, whether applied to paper requests or seed sets created to respond to them, could require disclosure of the effect of a narrowing position—such as a list of documents categorically withheld pursuant to an objection or the contours of a seed set in predictive coding, without the reasoning underpinning it. Though extending the same principle from objections to request interpretations (and associated excluded document populations from the collection) is a modest step, it has not been adopted as a rule.[12]

Perfect Recall of the Wrong Fact Population Is Useless

Even assuming a litigant did surface how the factual edges of a given document collection were determined, current eDiscovery disputes still largely center on costly debates about whether assertions of 70–95% recall or precision analysis of that collection are “accurate” or sufficient.[13] This not only misses the point about whether the document collection is the right one to examine, but also loses sight of what discovery is for: Discovery exists so that a requesting party can surface relevant facts to expedite just resolution of disputes,[14] not just records that predictably may or may not contain them. Even assuming litigants could agree that (a) one methodology over another will more reliably produce higher recall and precision for a given document collection and (b) certain recall and precision thresholds are per se proportional in a given case, that still doesn’t address the problem of whether a particular document collection could ever meet the information need of the parties.

As to solving that problem: Seasoned litigators do not need statistical sampling to be confident that at least some of the facts living outside a narrowly defined document collection may be critical to deciding a case and still others could be critical to deciding what a requesting party may see at all. Indeed, in modern litigation, disputes about the latter often serve as trench warfare for the power to prevail in the former.

The most relevant question in assessing whether a litigator is building such a trench with their chosen document collection definition is thus not “to what extent can a given document collection be accurately recalled and precisely analyzed,” but “which facts matter to an adjudicator’s task of determining whether the document collection was reasonably defined to capture the relevant document population in the first place?”

As every litigator representing a requesting party knows, the answer almost always comes from outside the disputed document collection, however it was originally defined: a payment stream in one party’s bank records suggesting the business records are incomplete; a message produced from one record custodian suggesting another custodian has the key information; a gap in a timeline that only makes sense if key records were moved or destroyed. In each case, the decisive evidence is not a machine model’s or a paralegal’s “recall” rate within a pile of records, but a litigant’s ability to draw inferences from contradictions between sources that each look clean or complete on their own.

A review process measuring only document-level analysis is blind to all of this. Such a review counts documents—and scores reviewers’ interpretation of them—inside one collection, while the facts that win discovery arguments more often live between collections.

The result is a system that, still very much like a game of Go Fish, requires proof of the hidden thing before allowing a look at it, without acknowledging the design of the game makes the necessary proof depend on facts the game itself never surfaces.

Humans Build the Decks: Facts Are Decided, Not Found

There is a deeper issue underpinning the Go Fish problems of TAR in the modern age, and it is the same one underpinning technology-assisted litigation as a whole: Whether litigation problems can be automated at all is not at base a question of how consistently a “deck” of records can be “proven” to contain relevant information, but whether any of us can “prove” a “fact”—either about a record in the deck (e.g., defining which cards are a “seven”) or contained in such a record (e.g., defining what it means to be a “seven”)—is a fact at all.

Attorneys dispute one another’s TAR protocols, not because anyone can prove one more validly arrives at the disputed “truth” of the matter than another, but because reasonable people, including “experts” in their fields, regularly weigh the same evidence about disputed matters differently.[15] This is, in fact, why the law assigns adjudication of disputed facts to neutral fact-finders: not because certain humans have omniscience in truth-seeing, but because in the face of widespread human ignorance about “truth,” someone must have authority to simply decide.

The critical pieces of our legal system are thus not a collective understanding of “ground truth” and confidence intervals about it, but shared beliefs that (1) a deliberately scrutable process for testing facts gets closer to truth than any alternative and (2) when challenged, the critical analysis is the rigor of the process of seeking truth (e.g., by asking if reasonable minds could disagree about “truth”), not truth itself.

Discovery in litigation has a structural oddity here. Though discovery battles often determine whether a litigant can win or lose at trial, they are still designed to happen before any neutral fact-finder is involved. Yet, when a producing party decides what a request covers, it is effectively adjudicating facts and making relevance determinations about them, alone, as an interested party.

If our legal system is truly designed to adjudicate the process of seeking truth, then even this first process should be visible enough to examine.

The Future of eDiscovery: Two Lures, Not One

The next generation of eDiscovery tools should not only strive to be more honest about the seal and shield creating stark information asymmetries among litigants, but also collapse two slow, separate fights in the Go Fish game—what to search and whether the search was adequate—into one faster process. That process requires two capabilities working together, and it is one Mary Technology is building today.

First, fact-based search on both sides. Machine-assisted review can already reach high precision and recall in document surfacing, and it can be tuned to many different readings of what makes a document collection rich.[16] The missing piece, as discussed above, is information symmetry. The missing tool to achieve that symmetry is not just one that can help litigants find and analyze records, but one that can help litigants reliably analyze facts, both within those records and in the negative space around them.

Imagine: The requesting party analyzes what it already holds and what has been produced, and submits corroborated facts for negative space analysis, rather than keyword lists. The producing party then runs fact-targeted queries against its own records and reports what they return, with support. Corroborated facts found on either side point both sides toward richer populations and collections faster. The point, to return to the Go Fish analogy, is not to analyze whether a draw from the deck is “accurate,” or to guess at what the other person drew, but to cooperate to define what could be in the deck in the first place.

Today, this could just be called “reasonable” discovery practice. In time, it could be expected or required.

Second, an inspectable record of interpretation. Every fact query embodies an interpretation of what is being asked. A sound process records that interpretation, why it was chosen, and what it might exclude, so the process itself can be examined later. Critically, this is not a log of machine decisions. It is a record of human decisions to delegate labor and fact analysis to machines: not just what was searched and analyzed, but how.

The second capability—which Mary Technology calls “productive friction”—is what keeps the system under human control. Because if interpretation of “facts” disappears into a model, no one can be held to account for it.[17] If the people who direct the model must state their interpretation before it runs, their judgment stays where the law should put it: with lawyers, and open to scrutiny and challenge.

Tools Built for Fact Discovery

Most eDiscovery review tools claim to optimize answers. A tool built for the new frontier must make the questions auditable.

First, as with existing tools, future tools must deliver high precision and recall to each user’s specifications. Second, they must help lawyers organize and analyze the facts within the records they surface.

To achieve the second goal, tool builders and lawyers alike must recognize that user specifications about optimal precision and recall are themselves empirical judgment calls about “truth” and the legal meaning of it. If humans are to remain the arbiters of both, modern eDiscovery tools cannot be calibrated to make those calls for attorneys or delegate them to agents. Rather, before a fact query runs, the tool should require the user to think through three things, and keep track of the results:

How the user interprets the query.

Why that interpretation matters to the case.

How the results could miss entire information populations that are equally rich, or richer.

Why? Because a discovery tool that forces attorneys to analyze factual queries and decide why and how to make them keeps the judgment at the heart of litigation fact analysis with human lawyers, where it belongs. More importantly, it leaves a record of those judgment calls that can be examined for adjudication later, by the team, by opposing counsel, or by a court.

Recall counts the cards. The future of eDiscovery is knowing who decided what counts as a seven.

Download the white paper as a PDF.

Notes

[1] Moore v. Publicis Groupe, 287 F.R.D. 182, 191 (S.D.N.Y. 2012), aff’d, No. 11 Civ. 1279 (ALC)(AJP), 2012 WL 1446534 (S.D.N.Y. Apr. 26, 2012) (citing Ralph C. Losey, Child’s Game of ‘Go Fish’ Is a Poor Model for e-Discovery Search, in Adventures in Electronic Discovery 209, 209–10 (2011)).

[2] Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125, 128 (S.D.N.Y. 2015) (“One TAR issue that remains open is how transparent and cooperative the parties need to be with respect to the seed or training set(s).”).

[3] Roshanak Omrani et al., Beyond the Bar: Generative AI as a Transformative Component in Legal Document Review n.5 (Relativity & Redgrave Data 2024) (preprint) (collecting authority that 70–85% recall is generally treated as reasonable and proportionate); see also Lawson v. Spirit AeroSystems, Inc., No. 18-1100-EFM-ADM, 2020 WL 1813395 (D. Kan. Apr. 9, 2020); In re Diisocyanates Antitrust Litig., No. MC 18-1001, 2021 WL 4295729 (W.D. Pa. Aug. 23, 2021), report and recommendation adopted, No. MC 18-1001, 2021 WL 4295719 (W.D. Pa. Sept. 21, 2021).

[4] Fed. R. Civ. P. 26(g)(1) (counsel certifies that disclosures and responses follow a “reasonable inquiry”).

[5] See William C. Dimm, Confirming Recall Adequacy with Unbiased Multi-Stage Acceptance Testing 1–3 (DESI VII 2017) (observing that “a recall estimate is not an adequacy determination,” and that sampling only the discard set makes results sensitive to shifts in the reviewer’s concept of responsiveness). Even leading judicial guidance overstates what these estimates establish. See Bolch Judicial Inst., Duke Law Sch., Technology Assisted Review (TAR) Guidelines 41 (Jan. 2019), https://judicialstudies.duke.edu/wp-content/uploads/2019/02/TAR-Guidelines-Final-1.pdf (stating that a 95% confidence level means “there is a 95% chance that the actual recall is between 75% and 85%,” a common misreading of what a confidence interval measures).

[6] See, e.g., Omrani et al., supra note 3, § 5.1 (acknowledging limitations on ground truth data available for evaluation), § 5.4 (gold standard of senior-attorney coding treated as “assumed correct”); Herbert L. Roitblat, FOMO: Topics Versus Documents in Legal eDiscovery (2020) § 6 (citing Roitblat, Kershaw, and Oot (2010): “human reviewers only agreed on 28% of the documents that either team found responsive”); Maura R. Grossman & Gordon V. Cormack, Inconsistent Assessment of Responsiveness in E-Discovery: Difference of Opinion or Human Error? 4–5, 9 (DESI IV 2011) (when a Topic Authority blindly re-reviewed ten of her own adjudications, she reaffirmed only five).

[7] Compare Syllo, Agentic AI Document Review Is Transformative for Complex Litigation 7–8 (Mar. 21, 2025) (reporting estimated recall of 100% where an elusion sample of the null set contained no responsive documents), with Dimm, supra note 5, at 2–3 (explaining why accept-on-zero elusion testing is biased and unreliable).

[8] See Rio Tinto, 306 F.R.D. at 128 (reasoning that seed-set transparency matters less when results can be checked through recall estimation and other outcome tests); Dynamo Holdings Ltd. P’ship v. Comm’r, Nos. 2685-11, 8393-12, 2016 WL 4204067 (T.C. July 13, 2016) (assuming the predictive coding response was flawed and still denying the motion to compel because the original inquiry was “reasonable”).

[9] See John M. Facciola & Philip J. Favro, Safeguarding the Seed Set: Why Seed Set Documents May Be Entitled to Work Product Protection, 8 Fed. Cts. L. Rev. 2, 6–8 & nn.40–42 (2015) (citing, e.g., Sporck v. Peil, 759 F.2d 312, 316 (3d Cir. 1985)).

[10] Fed. R. Civ. P. 26(b)(3)(B); Hickman v. Taylor, 329 U.S. 495 (1947); cf. Upjohn Co. v. United States, 449 U.S. 383, 395–96 (1981) (underlying facts are not protected merely because they were communicated to counsel).

[11] Fed. R. Civ. P. 34(b)(2)(C); see Fed. R. Civ. P. 34 advisory committee’s note to 2015 amendment.

[12] See, e.g., 7 Business and Commercial Litigation in Federal Courts § 80:9 (Robert L. Haig ed., 5th ed. 2025).

[13] See, e.g., id.

[14] See Fed. R. Civ. P. 1 (“These rules . . . should be construed . . . to secure the just, speedy, and inexpensive determination of every action and proceeding”); Hickman, 329 U.S. at 507 (“Mutual knowledge of all the relevant facts gathered by both parties is essential to proper litigation”); see also Roitblat, supra note 6 (describing eDiscovery as “the process of identifying responsive information, not identifying documents”).

[15] Cf. Grossman & Cormack, supra note 6, at 9–10 (concluding that most assessor disagreement reflects error, while finding that even an expert adjudicator’s own determinations did not hold up consistently on blind review).

[16] See Omrani et al., supra note 3, § 6 (protocol-prompted model achieved estimated 96% recall and 60% precision against senior-attorney coding); Syllo, supra note 7, at 2 (vendor-reported recall above 93% across ten live reviews).

[17] Particularly where the key “facts” are statistically complex, rare, or inherently unlearnable (i.e., not in a model’s training data) to start with. See Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala & Edwin Zhang, Evaluating Large Language Models for Accuracy Incentivizes Hallucinations, 653 Nature 1047, 1048 (2026), https://doi.org/10.1038/s41586-026-10549-w (finding that LLMs evaluated and selected for “closed-rubric accuracy” are incentivized to hallucinate); cf. Ralph Losey, Schrödinger’s Cat Goes to Court, Part Two: The Cat, the Gavel, and the New Age of Quantum Law, EDRM (Oct. 21, 2026), https://edrm.net/2026/10/schrodingers-cat-goes-to-court-part-two-the-cat-the-gavel-and-the-new-age-of-quantum-law/ (warning that a human reviewer who cannot understand a system’s assumptions, examine its validation record, or reject its recommendations provides only “human-in-the-loop theater,” not meaningful control).