Research

January 24 , 2024 by Sean Anderson

Legal Firms and Discovery

Discovery has been the cornerstone of legal practice for centuries, often largely relying on the retention of extremely well-caffeinated interns and legal aides. Even then, the human element of discovery and validation is not without its mishaps and confusion. This process of using humans to distill incredible amounts of information is costly, slow, and, I would argue, detrimental to their quality of life (you just can’t get back those hours!). 

In the last 15 years, e-discovery has made leaps in productivity, even while being based on legacy keyword search. Electronic discovery (also known as e-discovery, eDiscovery, or e-Discovery) is the electronic aspect of identifying, collecting, and producing electronically stored information (ESI) in response to a request for production in a lawsuit or investigation. E-discovery gives legal teams access to the entirety of their case content partnered with a library of external case law to be leveraged with a human at the wheel. This shift to online, highly searchable data made a large impact on the lives of legal teams and created an even more competitive market.

Evolving in these competitive markets is the use of generative AI to assist legal teams in performing discovery and sometimes even preparing case materials. Large Language Models (LLMs) give legal teams access to a huge amount of world training and information but very little, if any, information about your case or documents. Out-of-the-box LLMs simply can’t understand these specific domains. Enter, Retrieval Augmented Generation (RAG). With RAG, legal teams can conceptually and realistically ask questions of their specific data without having to hire a machine learning team and wait months to years for any utility. With RAG, there is no fine-tuning required. But even without the headache of tuning LLMs and building word class retrieval systems, many legal teams need to focus on their business, not becoming AI experts.

Importance of AI in the Legal Profession 

AI is shaping the future of legal work, both in the discovery and preparation of legal text. Organizations that are slow to adopt will find both the human capital and technical debt that they expose themselves to will far outweigh the risks. Data protection, security, and privacy need to be addressed. Luckily, there are modern solutions that harness the power of these large language models (LLMs) without exposing the model to training on your private case data. Once addressed, the benefits become largely clear and can include…

• Efficiency and Time-saving Benefits: search across seas of case material and existing case law with rapid iteration. Ask simple questions and get expansive answers.
• Enhanced Legal Research Capabilities: perform legal research and trial prep in a fraction of the time. Utilize your team resources for fact-checking and refinement vs. pure content generation.
• Automation of Routine Tasks: automate and operate at a larger scale. Utilize your team for creative and strategic tasks while still delivering in a capable and thorough manner
• Improved Decision-making and Case Analysis: go beyond the surface with deep knowledge and understanding. Reach across languages and folders to get the full picture of your case details.

These are only the current state of the market. Over time, you can expect AI to move from ANSWER into ACTION. The legal teams of the future will be able to do amazing things on behalf of their clients and in search of the truth. Key to delivering a future of action is tackling the most prominent barrier to widespread AI adoption: hallucinations.

Risks and Challenges

Every new technology comes with challenges, but arguably, the consequences in the legal profession are even more dire. Legal teams that want to evolve their use of AI need to be smart. This also means keeping privacy and data security front of mind. There are many AI tools on the market that allow you to hit the skids running quickly but offer you little data protection and no guarantees on how privacy will be handled. This means legal teams may need to deploy specialized solutions to address the specific requirements of the casework. In addition, there are specific areas with AI that still pose a significant risk to legal teams, should they be unable to navigate around them.

Hallucinations

Even the best models are plagued with hallucinations or inferences to incorrect or incomplete information. Often not given enough information, AI is prone to hallucination. RAG helps eliminate a majority of hallucination events, but it is not a foolproof solution. The best strategy is RAG combined with the ability to provide transparency in results and citations to allow humans to investigate further. With the right tools, legal teams can effectively navigate around the issues caused by occasional hallucinations.

One of the most topical examples of this was when a lawyer using a commercial airline used ChatGPT to write his court filings. Upon inspection, it was uncovered that ChatGPT had made up much of the information filed, which caused a federal judge to consider sanctions. It’s these kinds of series outcomes that have to be avoided.

Bias and Fairness Concerns in AI Algorithms 

There is a consistent concern that the training materials provided to an LLM can introduce new biases into legal discovery. There is also a conversation around fairness or even the availability and accessibility of training information. This is why, today it’s imperative to place a human-in-the-loop to ensure that AI results don’t cause harm. 

The implications of allowing bias into legal decision-making are also a focal concern. Legal teams need to operate on reliable and unbiased information to ensure the fidelity of their work is maintained while adopting these new technologies.

Ethical Considerations in AI Use 

The privacy and security implications of adopting generative AI are likely front of mind for many legal teams. The newsfeeds are littered with topical examples of how big tech companies have used and abused customer data. How can you make sure your team is effectively navigating these concerns but also not stifling your innovation?

Many popular open-source LLMs are unclear on their policies about training and access to end-user data. It is important that client privacy is addressed in any solution implemented by legal teams.

Similarly, it’s equally important that the AI solution provides transparency in its results and the ability to be accountable for the outcomes it produces. This is where the human-in-the-loop becomes so important, not as the source of all answers but as the mechanism to fact-check and verify the efficiency that AI produces.

Dependence on AI and Potential Job Displacement 

Let’s talk about the elephant in the room. It’s clear that AI will have a huge impact on legal discovery and drafting. It will greatly redefine the core and shake up the traditional legal roles. Many are afraid that this trend will go too far and that AI will replace legal experts. This may be true for base-level data entry and administrative tasks. However, as mentioned above, many AI systems still require a human-in-the-loop. Until the risk of hallucination is solved, there will always be a need for legal experts to review and verify data. There will be a need for legal professionals to upskill and adapt, and roles around data and querying will become even more differentiated and appreciated. A legal team’s ability to adapt to new technologies and implement them in a practical and useful way will ultimately decide their fate. We have some best practices for navigating AI adoption and avoiding common pitfalls.

Best Practices for AI Adoption in Legal Practice

Legal teams need to be smart about how they approach adopting AI, even more so than most organizations. Specific design patterns and best practices can be leveraged to ensure you are thinking comprehensively about mitigating the risks of implementing generative AI into your legal work.

Implement RAG to Better Tackle Hallucinations

Retrieval Augmented Generation (RAG) can be a helpful tool to ensure that answers are grounded in sound research. As a legal team, RAG offers a set of guardrails and controls that you can leverage to make sure you are getting a certain level of relevancy in your answers. RAG also drastically reduces the amount of hallucinations that plague many LLM models. By leveraging platforms that offer RAG-as-a-service, legal teams can leverage the benefits of RAG without having to be an ML or AI developer.

Ensuring Transparency and Explainability 

Tools and platforms that provide transparency and explainability in their answers should be leveraged. This might be something as simple as citations so an individual can audit the responses of a generative system. It also might take the form of open-source models that allow users to test the levels of hallucinations in the models they are using to manage the level of inherent hallucinations more effectively. Your team should aim to partner with an AI company that is dedicated to providing transparency in their responses, and not the consumer vendors who benefit from not addressing the security and privacy of customer data. Your AI platform should be a partner in providing you with this transparency, not a force keeping you from achieving it.

Human-in-the-loop Audits of AI Systems 

Generative AI platforms that provide citations and transparency allow humans to play a critical role in answer generation. Since no model is fully immune to hallucinations, it’s imperative that legal teams have the tools to be most productive when auditing AI results. Generative AI platforms that are truly developer-centric understand the role of the human auditor and provide them with the tools they need to focus on the areas most productive for human oversight.

Better Collaboration Between Legal Professionals and AI Developers

End-to-end generative AI platforms provide easy operation and bootstrapping of AI features to a wide range of users. Many of the more complex tasks still need to be handled by professionals with exposure to AI development or, at a minimum, some data science or machine learning knowledge. The impact of explainability of results means that both business users and technical teams can better collaborate on gaps they see in the AI solution. The ability for AI engineers to boil complex algorithms into strategic questions will help legal professionals with things like prompt engineering and refinements of interactive queries.

Continuous Education and Training for Legal Practitioners

It is imperative for legal teams to evolve their use of AI and implement continuous training and education due to the ever-evolving landscape of technology and its profound impact on the legal profession. AI tools and algorithms are constantly advancing, and staying up-to-date with these advancements is essential to harness their full potential. Continuous training ensures that legal professionals can effectively integrate AI into their practice, adapt to changing methodologies, and address emerging ethical and privacy concerns. By staying educated and evolving alongside AI technologies, legal teams can maintain their competitive edge, enhance the quality of their services, and better serve their clients while upholding the highest standards of legal practice.

Vectara and Legal Teams

Vectara is helping legal teams rise to the opportunity with a safe entry point for generative AI. Vectara is uniquely positioned to help legal teams because it never trains on your data and answers are specifically grounded in your data. This gives legal teams the ability to truly ask questions of their case data in a secure and private manner. Vectara does all this with an end-to-end hosted platform that removes the complexity of dealing with large language models. Any member of your team can use Vectara to quickly build question-and-answer systems without deep machine learning or AI expertise. At the same time, Vectara exposes users to the powerful features of generative AI and hybrid search to upskill your team’s understanding of the AI landscape. Vectara leverages Retrieval Augmented Generation to ground results in user data. This RAG also dramatically decreases the amount of hallucinations delivered. Users get full control over the significance of their results and what to share with other users. This is one of the most effective ways to reduce the amount of hallucinations that LLMs experience. 

Vectara also provides additional tools to give unprecedented transparency in AI results. In November, Vectara launched the Hughes Hallucination Evaluation Framework (HHEM) as an open-source model that anyone can use to test the number of hallucinations in not only Vectara’s platform but also for any open-source LLM. Vectara, in an effort to provide full transparency, also maintains a public leaderboard based on the model’s scoring.

By grounding answers in your own data, Vectara also reduces bias and balances fairness. This grounding ensures that LLMs raise their hand when they don’t fully understand something and points to areas where users may want to enrich information to get more reliable answers.

Vectara provides a very powerful yet accessible platform for building purpose-built question/answer systems. Don’t believe us? Check out our Legal Aid Demo, where you can ask questions about public data from www.illinoislegalaid.org, or the Enron demo, well, you know. It’s a great way to see Vectara’s retrieval and summarization in action over a huge data set. 

Conclusion

In conclusion, the adoption of AI in the legal field brings both promise and challenges. While AI models like RAG have made significant strides in reducing hallucinations and inaccuracies, they are not infallible. To effectively harness the power of AI, it is crucial to combine RAG with transparency measures that allow human oversight. Legal teams must be equipped with the right tools and practices to navigate the occasional pitfalls of AI while ensuring the integrity of their work.

Moreover, the concerns surrounding bias, fairness, and ethics in AI algorithms cannot be overlooked. The potential introduction of biases and the privacy implications of using generative AI demands careful consideration. Legal teams must prioritize protecting client privacy, demand transparency in AI results, and establish accountability for the outcomes produced. While AI may redefine traditional legal roles and impact job tasks, it is unlikely to replace legal experts entirely. The human-in-the-loop approach remains essential to verify data and mitigate the risks of AI-driven hallucinations. Ultimately, the ability of legal professionals to adapt, collaborate with AI developers, and continuously educate themselves will determine the successful adoption of AI in legal practice.

Vectara provides an end-to-end platform that gives legal teams a safe entry point into very powerful generative AI features while actively mitigating hallucinations and bias. Vectara can help any user or builder to construct question/answering systems like the one for Illinois Legal Aide. To learn more about how Vectara can help your legal team, visit our site.