A version of this article was first published in ThoughtLeaders4 FIRE Magazine.
As investigations become more complex and data grows exponentially, first-level document reviews become increasingly time-consuming and costly. As with any other technology-reliant process, artificial intelligence (AI) is being folded into document-review workflows to speed up the review phase and make investigations more efficient.
Until recently, the gold standard in a document review workflow has been the deployment of technology-assisted review (TAR), which uses technology alongside a human first-pass review. TAR involves a machine learning model that learns from reviewer-coding decisions and then uses that information to find and promote similar documents to the top of the review pile and potentially stop review when the rate of relevant documents drops to an acceptable level.
While a big step forward from the traditional search-term responsive batching that defined workflows in the past, the time it takes to review a large population of documents is significant.
Control Risks’ recent work with Relativity aiR for Review shows the benefits of using AI in document review.
Making light work of a heavy workload
A multinational with operations in Mexico recently partnered with Control Risks to conduct a sensitive internal investigation involving allegations of financial malfeasance and employee misconduct.
The case team faced a challenging workload constrained by a strict three-week deadline. More than 120,000 Spanish language documents required review and analysis for a report of key findings to be delivered to the client’s auditor and board. Using a standard review workflow would not have met the deadline without a significant (and disproportionate) uplift in first-level review costs.
With the agreement of the client and its legal team, Control Risks leveraged Relativity’s generative AI-powered document review solution, aiR for Review. The result was stark. Compared to a traditional first-level review approach, aiR for Review reduced the first-level review costs by approximately 50% and the total estimated review time by nearly 80%.
Let’s take a closer look at how the tool works.
Custom prompts for efficient document classification
Automation
Rather than undertaking a standard first-level human review, Control Risks and the case team used aiR for Review to automate the classification process by working alongside the case team to convert review protocol into detailed custom prompts. These prompts were based on the digestion of a case summary that included details such as key individuals, events, and specific concerns, as well as instructions to identify evidence of suspected wrongdoing.
Tagging
aiR for Review then used the prompts to apply predictive tagging to prioritise documents for second level human review. Relevance identified whether a document was pertinent to the investigation, while Issue assessed whether it related to a particular topic or legal concern. This allowed the team to quickly identify the most critical documents for further review, significantly speeding up the process.
(It is worth noting that aiR for Review looks at the extracted text of documents to make predictions. Investigation teams should be mindful of scanned, photographed, or handwritten notes, especially if Optical Character Recognition (OCR) is ineffective at capturing the text accurately as these may need to be reviewed in a separate workstream.)
Inputs
The effectiveness of aiR for Review relies on the quality of the prompt. Investigations teams must ensure that case details are captured in a way that the AI can process them effectively. Experienced investigators are crucial in crafting these prompts to avoid over-reliance on technology and ensure they identify nuanced details and correctly classify documents in complex cases. Thorough QC workflows must also be established to ensure the accuracy of predictions and iterate on prompts as necessary.
Iterative review and continuous learning
AI-powered tools like aiR for Review rely on human involvement to accurately classify documents.
In this case, Control Risks and the case team initially tested the custom prompts by running them over sample sets of documents. The review team manually assessed these samples to validate the classifications and make necessary adjustments, a crucial step that requires investigative expertise and knowledge of the intricacies of investigations to ensure accuracy and effectiveness. This iterative process allowed more thorough prompts to be written so that the AI system could better understand the nuances of the investigation.
Once the prompts were refined, the final iteration was run over the entire document set. This approach allowed investigators to focus their efforts on the most pertinent documents, streamlining the review process and ensuring that the team could meet the tight deadline. Prompt iteration and generating predictions over the data set was completed in four days.
Facilitating second-level (and subsequent) reviews
Documents flagged as key or highly relevant by aiR for Review were pushed to the top of the second-level review queue, ensuring that the most critical materials were addressed first. Running aiR for Review concurrently to identify specific issues helps to prioritise documents for second-level review by focusing on the most hot-button issues first.
The second-level review team also benefited from aiR for Review's generated comprehensive rationales, complete with citations for relevant documents, offering greater consistency than a team of first-level reviewers.
Foreign language optimisation
aiR for Review can review documents in a number of foreign languages and produce comprehensive review notes in English. Custom prompts can also be written in English regardless of the source language of the documents.
In this case, as the documents were primarily in Spanish, aiR for Review was instrumental in performing the first-pass review quickly and at a reduced cost by requiring a smaller team of Spanish-speaking reviewers. The initial sample sets were reviewed by a small team of Spanish speakers. The key or highly relevant documents were batched out directly to the case team for second-level review.
The future of AI in document review
The application of AI in document review is still evolving. As AI models become more sophisticated, their ability to understand and categorise complex documents will continue to advance. Future innovations will almost certainly include even more powerful algorithms that can perform deep semantic analysis, detect subtle anomalies in financial or transactional data, and integrate with other investigative tools to provide a more comprehensive review solution.
Progress in this space is rapid, and one key challenge for investigators and legal teams will be keeping up to date with it.