Product Brief

Predictive Coding from Theory to Practice: kCuras

Relativity Assisted Review
Date: October 2012 Author: Katey Wood, Analyst

Abstract: In a few short years, kCura has commandeered the service provider channel for e-discovery review,
leapfrogging industry behemoths in the market such as LexisNexis Concordance with aggressive pricing, agile
development, and superior usability in its Relativity review platform. With greater customer demand for predictive
coding, the company released Relativity Assisted Review in 2011, bringing a cutting-edge technology for expediting
attorney review to the fingertips of thousands of users. As adoption gathers steam, the company now enhances
usability to mainstream greater automation in review, the most expensive phase of e-discovery.

Overview
kCuras Relativity has garnered enviable adoption from service providers and, more recently, license sales, dominating
competition in a crowded marketplace for attorney review tools in e-discovery. The introduction of Relativity Analytics
added advanced search in recent years, and with the growth of computer-assisted review, some partners and power
users even built Relativitys extensible platform out on their own with workflow for predictive coding. With this cutting-
edge technology and workflow, attorney review can be partially automated by training a computer model to suggest
responsive documents based on machine learning from a pre-reviewed seed set. In order to mainstream the capability
with a court-defensible protocol, kCura released Relativity Assisted Review in 2011, featuring Six-Sigma-grade statistical
sampling and proprietary workflow.

Market Context
Predictive coding has floundered in mainstream adoption over the years, in part due to inertia from law firms in taking
on new technology or expediting review against the billable hour. But surging data volumes, more options for contract
or offshore legal services, and overall greater pricing pressure from corporate clients have increased the need for law
firms to offer more competitive rates. Alternative Fee Arrangements and itemized e-discovery costs topped the list of
corporate demands of law firms reported by respondents to a recent ESG research survey (see Figure 1). Moreover, 33%
of corporate counsel requested specific technology for attorney review in 2011, with 41% expecting to in 2012.1

1
Source: ESG Research Report, e-Discovery Market Trends: A View from the Legal Department, November 2011.

2012 by The Enterprise Strategy Group, Inc. All Rights Reserved.

 Product Brief: Predictive Coding from Theory to Practice 2

Figure 1. Corporate Counsel Requests from Outside Counsel

Have you requested any of the following measures of your outside counsel(s) over the
past 12 months? Which of the following do you plan to request within the next 12
months? (Percent of respondents) Planned
requests of
outside
Alternative fee arrangements 71%
counsel(s) as it
64%
pertains to
managing
Itemize e-discovery related fees 65% organizations
45% e-discovery
matters within
Purchase or use specific technology to assist with the next 12
41%
months (N=34)
document review 33%
Requested of
Purchase or use specific technology to assist with ESI pre- 38% outside
review and culling 30% counsel(s) as it
pertains to
managing
Purchase or use specific technology to assist with legal 24% organizations
hold notification and communication 21% e-discovery
matters over
Purchase or use specific technology to assist with case 26% the past 12
planning, preparation, risk and cost assessments 12% months (N=33)

0% 20% 40% 60% 80%

Source: Enterprise Strategy Group, 2012.

Recent court decisions have created a higher bar for defensibility and negotiation in the use of predictive coding
between parties, drawing attention to some high-profile horror stories. Yet users from law firms, service and staffing
providers, and even corporate counsel continue to wield predictive coding as a powerful weapon in their arsenals. As
mainstream providers such as kCura continue to drive prices down in the market, particularly the channel, consumers
weighing the cost/benefit of adopting the cutting-edge technology face a lower barrier to adoption. Could it be the first
wave of predictive coding mainstreaming?

Relativity Assisted Review

kCura released Relativity Assisted Review (RAR) in 2011. This adds to Relativitys existing four-year integration with
Content Analysts categorization engine for Latent Semantic Indexing, providing conceptual search and near-duplicate
detection (also known as Relativity Analytics). In order to support predictive coding, RAR features new defensible
workflow and Six-Sigma compliant random statistical sampling.

How it Works
RAR works by having attorneys train the engine with an initial coded seed set to identify other responsive documents
from the total collection, refining results until the system has reached an acceptable level of defensibility in retrieval.
The process requires two different user roles: domain experts who train the system and reviewers to code documents
separately as part of QC.
The initial training set of responsive documents can be selected in a few ways. Judgmental sampling using
acknowledged hot docs already determined to be responsive (identified as smoking guns or located with relevant
keywords) will find similar responsive documents. However, clustering and random statistical sampling can also identify
additional latent issues in the collectionthe unknown unknownsas an alternate or supplemental approach to avoid
missing potentially responsive issues that may not be initially apparent.
2012 by The Enterprise Strategy Group, Inc. All Rights Reserved.
 Product Brief: Predictive Coding from Theory to Practice 3

Once the initial training is completed, the machine classifies documents into three buckets: responsive, non-responsive,
and uncategorized. QC is then done by reviewers with a random sample of the documents already classified. The
system returns an overturn report of human-to-machine disagreement to record defects in the population. These
defect rates are used to determine whether the machine has reached an acceptable confidence level in classifying
documents.

Ensuring Success
The acceptable confidence level can be adjusted based on priorities for the matter; a lower confidence level with higher
tolerance for error may be acceptable in first pass review, reviewing production from opposing counsel, or other coarse-
grain tasks. In other cases, extensive QC by domain experts is required on machine-coded non-responsive documents to
make sure there is high recall and nothing relevant has been missed.
RAR is equipped to handle rolling document collections as new evidence enters a case. A Q4 2012 release will also
feature additional convenience features such as graphical reports and e-mail alerts to notify case admins of job
completion and other milestones. The tool does not feature hit highlighting of results to display for users what
prompted the model to promote a particular documenthence there is no indication of what might make it responsive.

Sales and Support

In the last year, kCura has transitioned pricing around the analytics offering; in addition to per-GB pricing, customers can
now purchase data at fixed rate ceilings for 1TB, 4TB, 25TB or unlimited data volume, as well as the option of unlimited
seats, with no incremental fees per case. Of course, partner-based sales depend on their individual pricing schemes.
Applied Discovery, for one, has already offered predictive coding for free in a limited-time offer; others such as Fios
provide unlimited analytics (including concept search) with any case in Relativity, but most often charge incremental
per-GB fees for Relativity Assisted Review projects.

The company claims more than 70 downloads of RAR, with over 80 terabytes of data in Relativity Analytics in the last
year. Adoption for its considerable channel of service provider partners is still developing; some early adopters offer
analytics capabilities on every case, the most active RAR partners being CDS, Fios, Applied Discovery, NightOwl, and Iris
Data. kCuras growing enterprise license deals with law firms and corporations have prompted even greater pick-up as a
percentage of sales.
kCura has made concerted efforts around support, which has consistently been an important differentiator for the
organization since it stakes its reputation on a strong channel, currently numbering 143 partners. RAR has its own
newsletter, is featured on the advice@kCura blog, and has spawned a webinar series.

Best Practices
RAR obviously has strong benefits for matters with large amounts of data which cannot easily be reviewed by humans,
such as M&A second requests. However, predictive coding is used in many scenarios, including internal matters,
bankruptcy, and civil litigation of all kinds, in particular he said-she said cases such as employment disputes. kCura
reports the inflection point is often in the 25,000-100,000 document range.
What other lessons learned or fail factors are involved? Users should be well educated on the tool going into
predictive coding, and consider the particulars of the case, the evidence at hand, and the overall team strategy.
Training and aligning the team beforehand sets expectations for using the tool. Defining seed set
documents to train the system is different from simply defining responsiveness; a document must be not
only responsive, but also a representative exemplar for training. The time, technique, and criteria for doing
this should be well understood going in, as should the likely outputs of the machine. Predictive coding
typically demands greater up-front investment than non-linear review, requiring expert review and
developed case strategy up-front rather than brute force document review for relevance or privilege.

2012 by The Enterprise Strategy Group, Inc. All Rights Reserved.

 Product Brief: Predictive Coding from Theory to Practice 4

Knowing the case and being consistent in execution is paramount. The strategy for preservation and
collection should match the downstream review strategy in terms of inclusiveness and need for
defensibility. In training the system, experts should be on the same page in defining responsiveness criteria
when tagging documents to avoid mixed signals. The best candidates for training the system may not even
be the subject matter expert in a case, but a project manager with the best knowledge of the data.
The data involved is key, particularly the presence or absence of known responsive documents within it. If
no smoking gun documents are known from the outset of a case, and the data collection is likewise not
rich in responsive documents, then it can be an exercise in patience to find them, resulting in linear review
of a lot of non-responsive evidence.

Competitive Landscape
Known by many namespredictive coding, suggested coding, intelligent prioritization, predictive tagging or ranking,
adaptive coding, AutoSuggest, Technology-Assisted Review, or Computer-Assisted Reviewgreater automation and
machine learning components have exploded in the attorney review market over the last year.
Recomminds Axcelerate Review and Analysis is an incumbent in the space, available as an enterprise license (as is its
collection and ECA tool), or on a hosted basis. Axcelerate is based on a similar statistical method of categorization to
RARs LSI, known as Probabilistic Latent Semantic Analysis. Clearwell (now owned by Symantec) released Transparent
Predictive Coding for its enterprise platform as well this year, presenting considerable competition both on the channel
side and in its strong traction with enterprise users. Autonomy (now owned by HP) also boasts the capability with
meaning-based coding. Other e-discovery and review tools claiming the capability include Catalyst Repositories,
Lateral Data (now owned by Xerox), and ZyLAB.
OEM-provider Equivio has been embedded in a number of review tools, while offering its own Relevance platform. For
clarification, kCura also has an ISV partnership with Equivio for near-duplicate detection and e-mail threading, but
Equivio is not the underlying engine of RAR. Orcatec, another OEM and platform provider, also has capabilities and
growing partnerships.
kCura has a substantial partner network, including Alix Partners, Applied Discovery, CDS, Deloitte, DiscoverReady, DTI,
EPIQ, Fios, H5, Huron Legal, Planet Data, RenewData, TechLaw Solutions, Teris, and TransPerfect, many of which offer
predictive coding either through RAR or another solution. Other service providers have embedded Content Analyst,
Equivio, or other IP and proprietary workflow into in-house review tools for predictive coding, offered as a service
these include Daegis, EPIQ, FTI, Kroll, Integreon, RenewData, Valora Technologies, and Xerox Litigation Services.

The Bigger Truth

In spite of common usage over the years by early adopters for large-scale projects such as Second Requests, the public
battle over predictive coding has continued over its court defensibility and usability by attorneys. kCuras measured
approach to sampling and typically comprehensive efforts at education and support for the tool should stand it in good
stead herealthough a lack of hit highlighting to demonstrate the logic behind automated decision-making could lead
to accusations of a black box, and concerns over novices deciphering outputs during training.
But in a broader sense, predictive coding will be won or lost based on price and tractionsomething kCura has been
known to tackle aggressively. Lowering the barrier to adoption around cost and availability for its partners and
enterprise clients will go a long way towards mainstreaming more automated review.
For one, the channel approach potentially sidesteps law firms who might be more resistant to adopting predictive
coding by putting it in the hands of service providers, clients, and legal staffing companies. Pairing this ready availability
with aggressive pricing (although pricing is likewise dependent on partners) additionally relieves potential users of a
painstaking cost/benefit thats often been the hidden cost of predictive codingis it worth it on this case, given that
Ill be charged a hefty per-GB fee to use it, I must re-train and re-tool my review processes, and Ill have to engage high-
charging attorneys first? Lastly, kCura is well-known and liked for its usability and superior user experience, giving it an
additional leg up. As predictive coding enters its land-grab phase of greater adoption, we expect Relativity to continue
setting the pace for competitors.
2012 by The Enterprise Strategy Group, Inc. All Rights Reserved.
 Product Brief: Predictive Coding from Theory to Practice 5

All trademark names are property of their respective companies. Information contained in this publication has been obtained by sources The Enterprise Strategy
Group (ESG) considers to be reliable but is not warranted by ESG. This publication may contain opinions of ESG, which are subject to change from time to time. This
publication is copyrighted by The Enterprise Strategy Group, Inc. Any reproduction or redistribution of this publication, in whole or in part, whether in hard-copy
format, electronically, or otherwise to persons not authorized to receive it, without the express consent of The Enterprise Strategy Group, Inc., is in violation of U.S.
copyright law and will be subject to an action for civil damages and, if applicable, criminal prosecution. Should you have any questions, please contact ESG Client
Relations at 508.482.0188.

2012 by The Enterprise Strategy Group, Inc. All Rights Reserved.