The art of RAG is evolving rapidly and as organizations look for the best solutions on the market, evaluating different RAG solutions for your specific needs has never been more critical. This Tuesday, we announced the release of Open RAG Eval, an open-source framework designed to provide transparent, efficient, and flexible evaluation of RAG implementations. Here, we aim to provide a quick overview and the benefits for why you should use Open RAG Eval.

Transparent, efficient, and flexible

Open RAG Eval is built with a commitment to open-source methodology, ensuring full transparency in calculating evaluation scores and how evaluation scores are calculated. Our goal is to encourage community-driven enhancements, allowing developers, researchers, and organizations to shape the tool’s evolution toward what the industry needs.

In collaboration with prominent researchers at the University of Waterloo, we’ve designed evaluation metrics that are both theoretically sound and practically applicable, ensuring that Vectara Eval meets the highest standards of rigor and usability.

Key features

Most existing evaluation tools, whether open-source or proprietary, require golden answers—a luxury many companies simply do not have. Businesses typically have raw data but lack predefined ideal responses for their specific use cases. Without golden answers, organizations often rely on subjective human evaluation or human-graded labeled sets, which are time-consuming, prone to human error, and difficult to scale. Open RAG Eval eliminates this limitation by providing a framework that allows users to measure performance objectively without requiring predefined answers.

Furthermore, Open RAG Eval empowers teams with automation to streamline the evaluation process. The tool is designed to be lightweight and easy to implement, making it accessible for both small teams and large enterprises. It enables organizations to test, iterate, and improve their search and RAG implementations with minimal overhead.

For enterprises that still require human evaluation, Open RAG Eval provides built-in support to incorporate human evaluation results efficiently. This allows organizations to quickly reupload the results and compare them against automated evaluations, bridging the gap between qualitative and quantitative assessment methods.

With flexible test case definition, customizable evaluation metrics, detailed reporting, and visualization, Open RAG Eval provides a robust and adaptable solution for optimizing and refining RAG solutions more effectively. It allows businesses to focus on data-driven improvements rather than spending excessive time on manual testing.

Metrics that matter

To ensure comprehensive evaluation, Open RAG Eval introduces the following cutting-edge metrics with the University of Waterloo:

UMBRELA – A holistic metric for assessing overall retrieval performance.

AutoNugget – Evaluates the presence of essential information nuggets from retrieved documents in generated responses.

Citation – Quantifies whether citations in the response are supported by the source documents.

Hallucination – Quantifies unsupported information in responses using Vectara’s Hallucination Evaluation Model.

By leveraging Open RAG Eval, you can:

• Build higher-quality AI assistants and search applications faster.
• Improve user satisfaction and engagement.
• Make data-driven decisions about RAGsearch optimization.
• Reduce time and effort spent on manual testing.

Join us in advancing RAG evaluation

We invite developers, researchers, and organizations to explore Open RAG Eval, contribute to its development, and help shape the future of RAG evaluation.

Together, let’s build a culture of open, community-built evaluation in search and AI-driven applications.

Get Started!Create your Vectara account today!Sign up for Vectara!