open-rag-eval RAG Evaluation without golden answers — Ofer Mendelevitch, Vectara
AI Summary
In this video, Offer from Victara presents ‘Open Rag Eval’, an innovative open-source project aimed at creating a quick and scalable solution for retrieval-augmented generation (RAG) evaluation. The project addresses the prevalent issue of relying on golden answers, which limits scalability. Developed in collaboration with the University of Waterloo, it features various connectors for data retrieval, including Vector, LangChain, and Lama Index. Key metrics for the evaluation process include the Umbrella metric for retrieval without golden chunks, the Auto Nuggetizer for generation assessment, citation faithfulness for evaluating citation accuracy, and hallucination detection. Additionally, viewers are introduced to a user-friendly interface for analysis of evaluation results. Offer encourages contributions to the project, highlighting its transparency and open-source nature.