{"id":7047,"date":"2018-03-21T14:06:49","date_gmt":"2018-03-21T18:06:49","guid":{"rendered":"http:\/\/jolt.richmond.edu\/?p=7047"},"modified":"2019-03-08T19:52:02","modified_gmt":"2019-03-09T00:52:02","slug":"john-henry-the-steam-drill-and-the-increasing-demands-of-rule-26g","status":"publish","type":"post","link":"https:\/\/blog.richmond.edu\/jolt\/2018\/03\/21\/john-henry-the-steam-drill-and-the-increasing-demands-of-rule-26g\/","title":{"rendered":"John Henry, the Steam Drill, and the Increasing Demands of Rule 26(g)"},"content":{"rendered":"

\"\"<\/a><\/p>\n

By: Kate Bauer,<\/p>\n

John Henry, well, he told his captain \/<\/em><\/p>\n

\u201cCaptain, a man, he ain’t nothing but a man \/<\/em><\/p>\n

Before I let your steam drill \/<\/em><\/p>\n

beat me down \/<\/em><\/p>\n

I’m gonna die with a hammer \/<\/em><\/p>\n

in my hand, Lord, Lord \/<\/em><\/p>\n

I’ll die with a hammer in my hand\u201d<\/em>[1]<\/a><\/p>\n

 <\/p>\n

John Henry cut through a mountain, exerting extraordinary effort to beat the steam drill.\u00a0 The price was steep, costing Henry his life.[2]<\/a> \u00a0Today new technologies providing reliable, cost-effective alternatives to routine human tasks seem to crop up daily.\u00a0 Innovators wield data to justify adopting the new technology; once adopted, technology\u2019s capabilities eventually become the standard against which human efforts are measured.\u00a0 Even superb workers struggle to compete with machines that do not eat, do not sleep, and do not tire.<\/p>\n

John Henry\u2019s story is currently playing out in the eDiscovery field.\u00a0 When identifying responsive documents for production, responding parties have a duty under Rule 26(g) to make a reasonable inquiry.[3]<\/a> \u00a0Requirements for a reasonable inquiry have been evolving in recent years.\u00a0 In the past, having humans manually review the documents was assumed to be accurate, with courts deferring to an attorney\u2019s professional judgment that he had made a reasonable inquiry unless the requesting party could show a deficiency.[4]<\/a>\u00a0 The increasing use of technology-assisted review (TAR)\u2014computer algorithms that analyze relevance decisions humans make on a small set of documents, and then extrapolate those decisions to the document collection[5]<\/a>\u2014has led to an increased emphasis on statistically validating the quality of the production set up front.[6]<\/a>\u00a0 These validation requirements were initially intended to objectively verify the accuracy of TAR algorithms, ensuring their results constituted a reasonable inquiry.[7]<\/a> \u00a0Now, at least one court has proposed requiring statistical validation of manual review as well, with threshold recommendations for what values constitute a \u201chigh-quality\u201d review.[8]<\/a>\u00a0 While research indicates that TAR can probably meet these requirements with ease, the accuracy demands will likely tax the capabilities of human reviewers.<\/p>\n

Humans, meet the steam drill.\u00a0 Good luck keeping up.<\/p>\n

\u00a0Reasonable Inquiry: Historically, a Deferential Standard<\/u><\/strong><\/p>\n

The Rules require an attorney to conduct a reasonable inquiry when putting together a production set.[9]<\/a> A reasonable production must be substantially complete, as \u201c[a]n evasive or incomplete disclosure, answer, or response must be treated as a failure to disclose, answer, or respond.\u201d[10]<\/a>\u00a0 To determine whether an inquiry is reasonable, the court looks to the totality of the circumstances.[11]<\/a> \u00a0The most common remedy for an inadequate production is a motion to compel, with sanctions generally reserved for when an attorney has failed to discover the obvious,[12]<\/a> when a party fails to timely disclose relevant information,[13]<\/a> or when a party obstructs access to relevant information.[14]<\/a> \u00a0Absent a showing of deficiency by the requesting party, courts treated a responding party\u2019s assertion of reasonableness deferentially.[15]<\/a><\/p>\n

To meet the \u201creasonable inquiry\u201d requirement, it was standard to hire armies of review attorneys to examine potentially relevant documents for privilege and production.[16]<\/a>\u00a0 Although utilizing massive review teams was generally considered reasonable if the team was adequately staffed and trained,[17]<\/a> the exhaustive review approach became increasingly unsustainable from a cost perspective as document volumes continued to grow.<\/p>\n

TAR: Reasonable Inquiry Requires Verification<\/u><\/strong><\/p>\n

At the turn of the 21st <\/sup>century, document discovery was an area ripe for innovation.\u00a0 The document review process was painfully expensive: a 1998 study of federal cases found that discovery amounted to at least half of all litigation expenses, and discovery represented 90% of total litigation costs in the most expensive cases.[18]<\/a>\u00a0 By 2014, most Fortune 1000 corporations were spending between $5 million and $10 million annually on eDiscovery costs, with seventy percent of those costs spent on document review.[19]<\/a>\u00a0 Several companies reported spending as much as $30 million on eDiscovery.[20]<\/a><\/p>\n

Innovators developed technology-assisted review to address the increasing data volume. \u00a0These computer algorithms promised to drastically reduce the time and cost of document discovery by \u201clearning\u201d from attorney decisions regarding relevance for a subset of documents, then applying the same designations to similar documents in the review set.[21]<\/a>\u00a0 Attorneys initially regarded these machine-learning claims with skepticism, so TAR advocates plied statistics to make the case that TAR was at least as accurate as exhaustive human review, and possibly more so.[22]<\/a>\u00a0 On top of its documented accuracy, on average TAR saved parties 45% of the costs of a traditional document review.[23]<\/a>\u00a0 Like the steam drill operators of old, small teams of specialists using TAR could replace the armies of human reviewers laying eyes on every document.[24]<\/a><\/p>\n

Today TAR advocates have largely succeeded: numerous courts have embraced TAR usage, albeit usually with caveats.[25]<\/a>\u00a0 In contrast to the historical deference to human judgment, when TAR was first adopted courts had concerns about its accuracy.\u00a0 Though academic studies had demonstrated impressive results for certain TAR solutions, the wide variety of solutions purporting to be TAR created fears about the \u201cblack box\u201d of the technology.[26]<\/a>\u00a0 That one TAR solution worked did not mean a different one would.\u00a0 Courts were therefore interested in objective statistical measurements such as precision and recall[27]<\/a> that would help them in determining whether the TAR process had worked (i.e.<\/em> whether it was reasonable).[28]<\/a> \u00a0Statistical validation allows parties to determine whether TAR is making good predictions or bad ones without verifying TAR\u2019s predictions for every document in the document set.[29]<\/a><\/p>\n

To ensure TAR was reasonable, courts encouraged parties to be more transparent with TAR productions.[30]<\/a>\u00a0 Rather than accepting an attorney\u2019s representation that he had conducted a reasonable inquiry, courts advocated (and at times required) that parties disclose unprecedented access detailing exactly how accurate their review had been.[31]<\/a> \u00a0This approach, though understandable, broke with the deferential approach courts had historically taken to manual review.<\/p>\n

Reevaluating Reasonable Inquiry: From Deference to Verification for Manual Review<\/u><\/strong><\/p>\n

The same statistics that supported TAR\u2019s superiority also exposed an uncomfortable truth: exhaustive human review is not the \u201cgold standard\u201d it had previously been assumed to be.[32]<\/a>\u00a0 In fact, research shows that manual human review is fraught with inconsistency: information retrieval professionals<\/em> disagree about whether a document is relevant at least as often as they agree.[33]<\/a> \u00a0Attorneys fare even worse: in one study, attorneys agreed on relevance, at best, a mere 28.1% of the time.[34]<\/a>\u00a0 Alternative methods to reduce document volume such as using search terms, or \u201ckeywords,\u201d also have significant shortcomings.[35]<\/a><\/p>\n

Nevertheless, before TAR\u2019s emphasis on statistical validation, little effort was expended to measure the accuracy of attorney coding decisions.[36]<\/a>\u00a0 Absent a showing to the contrary, courts accepted an attorney\u2019s professional judgment that a review was complete,[37]<\/a> which was embodied in the Rule 26(g) certification.[38]<\/a>\u00a0 A requesting party who disagreed had the burden of showing that the responding party\u2019s production was inadequate before a motion to compel would be granted.[39]<\/a><\/p>\n

The widespread usage of statistics to validate machine predictions has caused some to suggest applying a similar approach to manual review.\u00a0 As TAR became more mainstream, so too did awareness about the statistical metrics necessary to evaluate its performance.[40]<\/a> \u00a0The Rule 26(g) reasonableness inquiry is an objective one: what could be more objective than statistical validation?[41]<\/a><\/p>\n

Special Master Maura Grossman recently ordered parties to apply statistical validation to manual review in In Re Broiler Chicken Antitrust Litigation<\/em>, a class action currently pending in the Northern District of Illinois.[42]<\/a>\u00a0 Anticipating document-intensive discovery, Magistrate Judge Jeffrey Gilbert brought in Grossman, a well-known TAR expert, to arbitrate eDiscovery disputes.[43]<\/a>\u00a0 On January 3, 2018, Grossman issued a comprehensive Order governing processing, search methods, and validation.[44]<\/a>\u00a0 The Order observes that \u201c[t]he review process should incorporate quality-control and quality-assurance procedures to ensure a reasonable production consistent with the requirements of Federal Rule of Civil Procedure 26(g).\u201d[45]<\/a>\u00a0 Uniquely, the Order requires parties to use a Subject Matter Expert (SME) to calculate recall regardless of whether the parties use manual review or TAR<\/u>.[46]<\/a>\u00a0 While recognizing that recall alone is not dispositive, the Order states that \u201ca recall estimate on the order of 70% to 80% is consistent with, but not the sole indicator of, an adequate (i.e., high-quality) review.\u201d[47]<\/a><\/p>\n

The Broiler Chicken <\/em>validation provisions weigh heavily in favor of choosing TAR over manual review.\u00a0 Research shows that manual review tends to produce a wide range of recall values.[48]<\/a>\u00a0 For example, in their influential 2011 JOLT study Maura Grossman and Gordon Cormack compared the results of a TAR methodology against a manual review for five different review topics.[49]<\/a>\u00a0 While the TAR algorithms averaged a recall of 78.7% across all five categories (ranging from 67.3% to 86.5%), the human reviewers finished with an average recall of 59.3% (ranging from 25.2% to 79.9%).[50]<\/a>\u00a0 Similarly, in another study, researchers found that the recall of manual review ranged from 52.8% to 83.6%.[51]<\/a>\u00a0 Contrasting the wide range of manual review recall values with the tight band of TAR recall values supports the conclusion that manual review runs a greater risk of failing the Broiler Chicken<\/em> recall validation protocol.<\/p>\n

In addition to the increased risk of failing the Broiler Chicken<\/em> validation protocol, manual human review involves a costly level of effort. While Rule 1 asserts that the Rules \u201cshould be construed, administered, and employed . . . to secure the just, speedy, and inexpensive determination of every action and proceeding,\u201d[52]<\/a> purely human efforts generally involve greater time and expense than TAR.[53]<\/a>\u00a0 As shown by the JOLT study, the increased time and expense of human review is unlikely to translate into greater accuracy than TAR provides.[54]<\/a><\/p>\n

Electing human review over TAR will usually run afoul of Rule 1\u2019s objectives to increase the speedy determination of actions while reducing costs and preserving justice.[55]<\/a>\u00a0 Most obviously, manual review takes longer and incurs higher costs that TAR.[56]<\/a>\u00a0 Manual review is usually billed at a per-reviewer hourly rate, or on a per-document basis across the entire corpus of potentially relevant documents.[57]<\/a>\u00a0 TAR, by contrast, only requires human review of a small subset of documents, with the technology eliminating the need to manually review large portions of the document set.[58]<\/a>\u00a0 Additionally, when more reviewers are used, there is a greater chance that their interpretations of relevance will vary, increasing the risk of low recall.[59]<\/a>\u00a0 In contrast, a small number of SMEs using TAR to extrapolate their judgments will likely have a more consistent view of relevance than dozens of reviewers.[60]<\/a> \u00a0Lastly, if the recall of the manual review is not at an acceptable level, the existing coding must be manually reevaluated on a document-by-document basis\u2014a laborious and costly process.\u00a0 However, correcting a TAR algorithm\u2019s incorrect predictions is simple: just change the coding designation on the document the algorithm used to make its prediction.\u00a0 Within hours (if not minutes) the algorithm will adjust its predictions for all related documents in the document set.\u00a0 Not even the John Henrys of the document review world can compete with such speed.<\/p>\n

Broiler Chicken<\/u><\/em><\/strong>: A Pecking Order for the Future? <\/u><\/strong><\/p>\n

Does Broiler Chicken<\/em> portend a shift in how courts will assess the Rule 26(g) reasonable inquiry going forward?\u00a0 Though the Order permits parties to choose either manual review or TAR, there is little doubt that the validation protocol favors TAR usage.\u00a0 Academic studies and in-the-field usage have established that using TAR is less expensive and faster than manual review, while being at least as accurate (if not more so).[61]<\/a>\u00a0 A party who chooses manual review risks (1) falling short of the accuracy mark after incurring substantial time and expense, and (2) having to reevaluate its coding decisions one-by-one.<\/p>\n

TAR has not yet entirely surpassed human capabilities, but that time may be coming. \u00a0In the Grossman and Cormack JOLT study, on one topic the humans did outperform the algorithm.[62]<\/a>\u00a0 Though less than 1% separated man from machine,[63]<\/a> this slim win for human accuracy shows that traditional methods are not yet obsolete.\u00a0 For now, courts generally accept that \u201c[r]esponding parties are best situated to evaluate and select the procedures, methodologies, and technologies for their e-discovery process.\u201d[64]<\/a>\u00a0 Still, the time may be coming when \u201cit might be unreasonable for a party to decline to use TAR.\u201d[65]<\/a>\u00a0 After all, John Henrys are rare, but the technology to rival him is cheap, fast, and readily available.<\/p>\n

 <\/p>\n

[1]<\/a> Bruce Springsteen with the Seeger Sessions Band, BBC Four Sessions (2008) (covering Pete Seeger\u2019s song \u201cJohn Henry\u201d), https:\/\/www.youtube.com\/watch?v=U3eutnpTr3E.<\/p>\n

[2]<\/a> For an alternative theory, see<\/em> William Grimes, Taking Swings at a Myth, With John Henry the Man<\/em>, N.Y. Times (Oct. 18, 2006) (\u201cA smoothly coordinated human team had an advantage over the early drills, which constantly broke down.\u00a0 The machines were highly efficient, however, at generating clouds of silicon dust. Contrary to the picture presented by the ballads, John Henry would have died not of exhaustion or a burst heart, but of silicosis, a fatal, fast-moving lung disease that took the lives of hundreds of railroad workers.\u201d).<\/p>\n

[3]<\/a> Fed. R. Civ. P. 26(g).<\/p>\n

[4]<\/a> See <\/em>Herbert L. Roitblat, The Pendulum Swings: Practical Measurement in eDiscovery<\/em>, OrcaBlog (Nov. 4, 2014, 3:57 PM), https:\/\/web.archive.org\/web\/20150405030235\/http:\/\/orcatec.com\/2014\/11\/04\/the-pendulum-swings-practical-measurement-in-ediscovery\/ (\u201cWe have gone from assessing eDiscovery on the basis of an attorney\u2019s opinion: \u2018I\u2019m a professional and I conducted a professional enquiry,\u2019 to . . . a perceived need to \u2018prov[e] that we have obtained a certain level of recall.\u2019\u201d); see also<\/em> Hyles v. New York City, No. 10CIV3119ATAJP, 2016 WL 4077114 at *3 (S.D.N.Y. Aug. 1, 2016); Ford Motor Co. v. Edgewood Properties, Inc., 257 F.R.D. 418, 427\u201328 (D.N.J. 2009); The Sedona Conference, The Sedona Principles, Third Edition: Best Practices, Recommendations & Principles for Addressing Electronic Document Production<\/em>, 19 Sedona Conf. J. 1, 52 (2018) (\u201cThe requesting party has the burden on a motion to compel to show that the responding party\u2019s steps to preserve and produce relevant electronically stored information were inadequate.\u201d).<\/p>\n

[5]<\/a> Maura R. Grossman & Gordon V. Cormack, The Grossman-Cormack Glossary of Technology-Assisted Review<\/em>, 7 Fed. Cts. L. Rev. 1, 32 (2013) (Technology-assisted review is a \u201cprocess for Prioritizing or Coding a Collection of Documents using a computerized system that harnesses human judgments of one or more Subject Matter Expert(s) on a smaller set of Documents and then extrapolates those judgments to the remaining Document Collection.\u201d).<\/a><\/p>\n

[6] See<\/em> Roitblat, supra<\/em> at note 4.<\/p>\n

[7]<\/a> See<\/em> Da Silva Moore v. Publicis Groupe & MSL Grp., 287 F.R.D. 182, 192 (S.D.N.Y. 2012) (\u201c[I]t is unlikely that courts will be able to determine or approve a party’s proposal as to when review and production can stop until the computer-assisted review software has been trained and the results are quality control verified. Only at that point can the parties and the Court see where there is a clear drop off from highly relevant to marginally relevant to not likely to be relevant documents.\u201d).<\/a><\/p>\n

[8] In re Broiler Chicken Antitrust Litig., No. 1:16-cv-08637, 2018 U.S. Dist. LEXIS 33140 at *50\u201351 (N.D. Ill. Jan. 3, 2018).<\/p>\n

[9]<\/a> Fed. R. Civ. P. 26(g).<\/p>\n

[10]<\/a> See<\/em> Fed. R. Civ. P. 37(a)(4).<\/p>\n

[11]<\/a> Fed. R. Civ. P. 26 advisory committee notes to the 1983 amendments.<\/p>\n

[12]<\/a> R & R Sails Inc. v. Ins. Co. of Penn., 251 F.R.D. 520, 525 (S.D. Cal. 2008).<\/p>\n

[13]<\/a> Gucci America, Inc. v. Costco Wholesale, No. 02 Civ. 3190 (DAB) (RLE), 2003 WL 21018832 at *2 (S.D.N.Y. May 6, 2003).<\/a><\/p>\n

[14] St. Paul Reinsurance Co. v. Commercial Financial Corp., 198 F.R.D. 508, 511 (N.D. Iowa 2000).<\/p>\n

[15]<\/a> See<\/em> The Sedona Conference, supra <\/em>note 4, at 52.<\/a><\/p>\n

[16] See, e.g.<\/em>, Datel Holdings Ltd. v. Microsoft Corp., No. C-09-05535 EDL, 2011 WL 866993 at *4 (N.D. Cal. Mar. 11, 2011) (finding that party took reasonable steps to prevent disclosure of privileged documents including providing review attorneys with written instructions and a tutorial for the review); Kandel v. Brother Int\u2019l Corp., 683 F. Supp. 2d 1076, 1085\u201386 (C.D. Cal. 2010) (finding that party had taken reasonable steps to prevent disclosure of privileged documents by staffing and training a document-review team).<\/a><\/p>\n

[17] See<\/em> Datel, 2011 WL 866993 at *4; Kandel<\/em>, F. Supp. 2d at 1085\u201386.<\/p>\n

[18]<\/a> Scott A. Moss, Litigation Discovery Cannot Be Optimal but Could Be Better: The Economics of Improving Discovery Timing in A Digital Age<\/em>, 58 Duke L.J. 889, 892 (2009).<\/p>\n

[19]<\/a> Jennifer Booton, Don\u2019t Send Another Email Until You Read This<\/em>, MarketWatch (Mar. 9, 2015, 10:10 AM), http:\/\/www.marketwatch.com\/story\/your-work-emails-are-now-worth-millions-of-dollarsto-lawyers-2015-03-06.<\/a><\/p>\n

[20] Id.<\/em><\/a><\/p>\n

[21] See<\/em> Grossman & Cormack, supra<\/em> note 5.<\/a><\/p>\n

[22] See <\/em>Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review<\/em>, XVII Rich. J.L. & Tech. 1, 37 (2011) (reporting that manual reviewers identified between 25% and 80% of relevant documents, while technology-assisted review returned between 67% and 86%); see also<\/em> Herbert L. Roitblat et al., Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review<\/em>, J. of Am. Soc\u2019y for Info.\u00a0 Sci. & Tech. 70, 79 (2010) (performing an empirical assessment to \u201canswer the question of whether there was a benefit to engaging in a traditional human review or whether computer systems could be relied on to produce comparable results,\u201d and concluding that \u201c[o]n every measure, the performance of the two computer systems was at least as accurate (measured against the original review) as that of human re-review.\u201d).<\/p>\n

[23]<\/a> eDiscovery Institute Survey on Predictive Coding<\/em>, eDiscovery Institute 3 (Oct. 21, 2010), https:\/\/www.discovia.com\/wp-content\/uploads\/2012\/07\/2010_EDI_PredictiveCodingSurvey.pdf.<\/a><\/p>\n

[24] Anne Kershaw & Joseph Howie, Crash or Soar? Will the Legal Community Accept \u201cPredictive Coding?\u201d<\/em>, L. Tech News, Oct. 2010, http:\/\/www.knowledgestrategysolutions.com\/wp-content\/uploads\/LTN_CrashOrSoar_2010_Oct.pdf.<\/p>\n

[25]<\/a> See, e.g.<\/em>, Green v. Am. Modern Home Ins. Co., No. 14\u2013CV\u201304074, 2014 WL 6668422 at *1 (W.D. Ark. Nov. 24, 2014); Dynamo Holdings Ltd. P\u2019Ship v. Comm\u2019r of Internal Revenue, 143 T.C. 183, 185 (T.C. Sept. 17, 2014); Aurora Coop. Elevator Co. v. Aventine Renewable Energy\u2013Aurora W. LLC, No. 12 Civ. 0230, Dkt. No. 147 (D. Neb. Mar. 10, 2014); Edwards v. Nat\u2019l Milk Producers Fed\u2019n, No. 11 Civ. 4766, Dkt. No. 154: Joint Stip. & Order (N.D. Cal. Apr. 16, 2013); Bridgestone Am., Inc. v. IBM Corp., No. 13\u20131196, 2014 WL 4923014 (M.D. Tenn. July 22, 2014); Fed. Hous. Fin. Agency v. HSBC N.A. Holdings, Inc., 11 Civ. 6189, 2014 WL 584300 at *3 (S.D.N.Y. Feb. 14, 2014); EORHB, Inc. v. HOA Holdings LLC, No. Civ. A. 7409, 2013 WL 1960621 (Del. Ch. May 6, 2013); In re Actos (Pioglitazone) Prods. Liab. Litig, No. 6:11\u2013MD\u20132299, 2012 WL 7861249 (W.D. La. July 27, 2012) (Stip. & Case Mgmt. Order); Global Aerospace Inc. v. Landow Aviation LP, No. CL 61040, 2012 WL 1431215 (Va. Cir. Ct. Apr. 23, 2012); Da Silva Moore v. Publicis Groupe & MSL Grp., 287 F.R.D. 182 (S.D.N.Y. 2012).<\/a><\/p>\n

[26] Andrew Peck, Search, Forward<\/em>, L. Tech. News, Oct. 2011, 25, 29 (\u201c[I]f the use of predictive coding is challenged in a case before me, I will want to know what was done and why that produced defensible results. I may be less interested in the science behind the \u2018black box\u2019 of the vendor’s software than in whether it produced responsive documents with reasonably high recall and high precision.\u201d).<\/p>\n

[27]<\/a> To calculate recall, a party must review a mix of relevant and irrelevant documents.\u00a0 The party then codes the documents for relevance and compares its decisions about relevance against the algorithm\u2019s predictions about relevance.\u00a0 The more alignment between human reviewer and the algorithm\u2019s predictions, the better the recall. See<\/em> Grossman & Cormack, supra<\/em> note 5 at 27 (defining \u201crecall\u201d as \u201c[t]he fraction of Relevant Documents that are identified as Relevant by a search or review effort.\u201d).<\/p>\n

[28]<\/a> Da Silva Moore, 287 F.R.D. at 185 (\u201cThe Court explained that \u2018where [the] line will be drawn [as to review and production] is going to depend on what the statistics show for the results,\u2019 since \u2018[p]roportionality requires consideration of results as well as costs.\u2019\u201d).<\/p>\n

[29]<\/a> John Tredennick, Ask Catalyst: In TAR, What Is Validation And Why Is It Important?<\/em>, Catalyst (Sept. 27, 2016), https:\/\/catalystsecure.com\/blog\/2016\/09\/ask-catalyst-in-tar-what-is-validation-and-why-is-it-important\/.<\/a><\/p>\n

[30] See id.<\/em>; see also<\/em> Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125, 128\u201329 (S.D.N.Y. 2015); Bridgestone, 2014 U.S. Dist. WL 4923014 at *1 (\u201c[O]penness and transparency in what Plaintiff is doing will be of critical importance. Plaintiff has advised that they will provide the seed documents they are initially using to set up predictive coding. The Magistrate Judge expects full openness in this matter.\u201d); Progressive Cas. Ins. Co.<\/em>, 2014 U.S. Dist. WL 3563467 at *11 (declining to allow predictive coding when counsel was \u201cunwilling to engage in the type of cooperation and transparency that . . . is needed for a predictive coding protocol to be accepted by the court . . . .\u201d); Transcript of Record at 9, 14, Fed. Hous. Fin. Agency v. JPMorgan Chase & Co., No. 1:11-cv-06188 (S.D.N.Y. July 24, 2012) (bench decision requiring transparency and cooperation, including giving the plaintiff full access to the seed set’s responsive and non-responsive documents except privileged).<\/a><\/p>\n

[31] See <\/em>sources cited supra<\/em> note 30.<\/a><\/p>\n

[32] See<\/em> The Sedona Conference, The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery<\/em>, 8 Sedona Conf. J. 189, 199 (2007) (\u201c[T]here appears to be a myth that manual review by humans of large amounts of information is as accurate and complete as possible– perhaps even perfect–and constitutes the gold standard by which all searches should be measured.\u201d).<\/p>\n

[33]<\/a> Ellen M. Voorhees, Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness<\/em>, 36 Info. Processing & Mgmt. 697, 701 (2000) (concluding that assessors disagree that a document is relevant at least as often as they agree).<\/p>\n

[34]<\/a> Roitblat et al., supra<\/em> note 22, at 74.<\/p>\n

[35]<\/a> David C. Blair & M.E. Maron, An Evaluation of Retrieval Effectiveness for a Full-Text Document-Retrieval System<\/em>, 28 Commc\u2019ns Ass\u2019n Computing Mach. 289, 295\u201396 (1985) (finding that paralegals who thought they had retrieved 75% of relevant documents using iterative keyword searches had only found 20%).<\/p>\n

[36]<\/a> Roitblat, supra<\/em> note 4.<\/a><\/p>\n

[38]<\/a> See<\/em> sources cited supra<\/em> note 4.<\/a><\/p>\n

[39] See<\/em> sources cited supra<\/em> note 4.<\/a><\/p>\n

[40] See<\/em> John Tredennick, Measuring Recall in E-Discovery Review, Part Two: No Easy Answers<\/em>, Catalyst (Dec. 5, 2014), https:\/\/catalystsecure.com\/blog\/2014\/12\/measuring-recall-in-e-discovery-review-part-two-no-easy-answers\/; Ralph Losey, Visualizing Data in a Predictive Coding Project<\/em>, e-Discovery Team (Nov. 9, 2014, 8:10 PM), https:\/\/e-discoveryteam.com\/2014\/11\/09\/visualizing-data-in-a-predictive-coding-project\/; Herbert L. Roitblat, The Pendulum Swings: Practical Measurement in eDiscovery<\/em>, OrcaBlog (Nov. 4, 2014, 3:57 PM), https:\/\/web.archive.org\/web\/20150405030235\/http:\/\/orcatec.com\/2014\/11\/04\/the-pendulum-swings-practical-measurement-in-ediscovery\/; John Tredennick, Measuring Recall in E-Discovery Review, Part One: A Tougher Problem Than You Might Realize<\/em>, Catalyst (Oct. 15, 2014), https:\/\/catalystsecure.com\/blog\/2014\/10\/measuring-recall-in-e-discovery-review-a-tougher-problem-than-you-might-realize-part-1\/.<\/p>\n

[41]<\/a> Too bad for the people who became lawyers because they were told there would be no math.<\/a><\/p>\n

[42] In re Broiler Chicken Antitrust Litig., No. 1:16-cv-08637, 2018 U.S. Dist. LEXIS 33140 (N.D. Ill. Jan. 3, 2018).<\/a><\/p>\n

[43] Michele C. S. Lange, TAR Protocol Rules the Roost: In Re Broiler Chicken<\/em>, The ACEDS eDiscovery Voice (Feb. 8, 2018), http:\/\/www.aceds.org\/blogpost\/1653535\/294460\/TAR-Protocol-Rules-the-Roost-In-Re-Broiler-Chicken.<\/a><\/p>\n

[44] Broiler Chicken, No. 1:16-cv-08637, 2018 U.S. Dist. LEXIS 33140 at *45, *47 (N.D. Ill. Jan. 3, 2018).<\/a><\/p>\n

[45] Id.<\/em> at *45.<\/a><\/p>\n

[46] Id.<\/em> at *45, *47.<\/p>\n

[47]<\/a> Id.<\/em> at *50\u201351.<\/a><\/p>\n

[48] See<\/em> Grossman & Cormack, supra<\/em> note 22 at 37; Roitblat et al., supra<\/em> note 22 at 79.<\/a><\/p>\n

[49] Grossman & Cormack, supra<\/em> note 22 at 37.<\/a><\/p>\n

[50] Id.<\/em><\/p>\n

[51]<\/a> Roitblat et al., supra<\/em> note 22, at 79.<\/p>\n

[52]<\/a> Fed. R. Civ. P. 1.<\/p>\n

[53]<\/a> See<\/em> eDiscovery Institute Survey on Predictive Coding<\/em>, eDiscovery Institute, ii, 3 (Oct. 21, 2010), https:\/\/www.discovia.com\/wp-content\/uploads\/2012\/07\/2010_EDI_PredictiveCodingSurvey.pdf.<\/p>\n

[54]<\/a> See<\/em> Grossman & Cormack, supra<\/em> note 22 at 37; Roitblat et al., supra<\/em> note 22 at 79.<\/p>\n

[55]<\/a> See<\/em> Fed. R. Civ. P. 1.<\/p>\n

[56]<\/a> See<\/em> eDiscovery Institute, supra<\/em> note 53.<\/p>\n

[57]<\/a> See, e.g.<\/em>, Patrick Oot et. al., Mandating Reasonableness in A Reasonable Inquiry<\/em>, 87 Denv. U. L. Rev. 533, 548 (2010) (\u201c[To review 1.6 million documents] [i]t took the attorneys four months, working sixteen hours per day seven days per week, for a total cost of $13,598,872.61 or about $8.50 per document.\u201d).<\/p>\n

[58]<\/a> See<\/em> Grossman & Cormack, supra<\/em> note 5.<\/p>\n

[59]<\/a> See <\/em>Roitblat et al., supra<\/em> note 22, at 74.<\/p>\n

[60]<\/a> See <\/em>Voorhees, supra<\/em> note 33.<\/p>\n

[61]<\/a> See<\/em> Grossman & Cormack, supra<\/em> note 22, at 37; Roitblat et al., supra<\/em> note 22, at 79.<\/a><\/p>\n

[62] Grossman & Cormack, supra<\/em> note 22, at 37.<\/p>\n

[63]<\/a> Id. <\/em><\/a><\/p>\n

[64] See, e.g.<\/em>, Hyles v. New York City, No. 10CIV3119ATAJP, 2016 WL 4077114 at *3 (S.D.N.Y. Aug. 1, 2016); Kleen Prod. LLC v. Packaging Corp. of Am., No. 10 C 5711, 2012 WL 4498465 at *5 (N.D. Ill. Sept. 28, 2012); Ford Motor Co. v. Edgewood Properties, Inc., 257 F.R.D. 418, 427 (D.N.J. 2009); see also<\/em> The Sedona Conference, supra <\/em>note 4, at 52.<\/p>\n

[65]<\/a> Hyles, 2016 WL 4077114 at *3.<\/p>\n

Image Source:\u00a0https:\/\/www.nrm.org\/2016\/06\/john-henry-2009\/.<\/p>\n","protected":false},"excerpt":{"rendered":"

By: Kate Bauer, John Henry, well, he told his captain \/ \u201cCaptain, a man, he ain’t nothing but a man \/ Before I let your steam drill \/ beat me down \/ I’m gonna die with a hammer \/ in my hand, Lord, Lord \/ I’ll die with a hammer in my hand\u201d[1]   John […]<\/p>\n","protected":false},"author":4287,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[51366],"tags":[],"class_list":["post-7047","post","type-post","status-publish","format-standard","hentry","category-blog-post"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/paMHOZ-1PF","jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/blog.richmond.edu\/jolt\/wp-json\/wp\/v2\/posts\/7047","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.richmond.edu\/jolt\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.richmond.edu\/jolt\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.richmond.edu\/jolt\/wp-json\/wp\/v2\/users\/4287"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.richmond.edu\/jolt\/wp-json\/wp\/v2\/comments?post=7047"}],"version-history":[{"count":0,"href":"https:\/\/blog.richmond.edu\/jolt\/wp-json\/wp\/v2\/posts\/7047\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.richmond.edu\/jolt\/wp-json\/wp\/v2\/media?parent=7047"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.richmond.edu\/jolt\/wp-json\/wp\/v2\/categories?post=7047"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.richmond.edu\/jolt\/wp-json\/wp\/v2\/tags?post=7047"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}