Building, Evaluating, and Optimizing your RAG App for Production | Jerry Liu

Name: Building, Evaluating, and Optimizing your RAG App for Production | Jerry Liu
Uploaded: 2023-12-01
Duration: 19 min
Description: LlamaIndex co-founder Jerry Liu on taking a RAG system from a five-line prototype to production: the two-stage stack and the common failure modes.

Generative AI
Enterprise h2oGPTe

Overview

In this H2O World session, Jerry Liu, co-founder and COO of LlamaIndex, walks through how to move a retrieval augmented generation (RAG) system from a five line prototype to a production ready application. He breaks down the two stage RAG stack, data ingestion and querying, the most common failure modes in naive RAG, bad retrieval and bad generation, and how to evaluate systems using both retrieval specific and end to end metrics. The session closes with a tour of optimization levers, from chunk sizing and prompt engineering to embedding model choice and small to big retrieval.

Key takeaways

The basic RAG stack: data ingestion (parsing, chunking, and embedding into a vector database) versus querying (retrieval plus synthesis)
The two core failure modes of naive RAG: bad retrieval (low precision, low recall, outdated data) and bad response generation (hallucination, lost context, toxicity or bias)
How to evaluate RAG systems both in isolation (retrieval only, using ranking metrics like NDCG and precision at k) and end to end (query and response datasets scored with LLM based evaluators)
Table stakes optimization levers: chunk size tuning, reranking, prompt engineering, and few shot examples
Why the choice of LLM and embedding model both have an outsize impact on performance, and how to benchmark them against your own data

Start owning your AI

One platform. Complete sovereignty. Measurable results.

Request a demo Talk to H2O