Building, Evaluating, and Optimizing your RAG App for Production | Jerry Liu
- Generative AI
- Enterprise h2oGPTe
Overview
In this H2O World session, Jerry Liu, co-founder and COO of LlamaIndex, walks through how to move a retrieval augmented generation (RAG) system from a five line prototype to a production ready application. He breaks down the two stage RAG stack, data ingestion and querying, the most common failure modes in naive RAG, bad retrieval and bad generation, and how to evaluate systems using both retrieval specific and end to end metrics. The session closes with a tour of optimization levers, from chunk sizing and prompt engineering to embedding model choice and small to big retrieval.
Key takeaways
- The basic RAG stack: data ingestion (parsing, chunking, and embedding into a vector database) versus querying (retrieval plus synthesis)
- The two core failure modes of naive RAG: bad retrieval (low precision, low recall, outdated data) and bad response generation (hallucination, lost context, toxicity or bias)
- How to evaluate RAG systems both in isolation (retrieval only, using ranking metrics like NDCG and precision at k) and end to end (query and response datasets scored with LLM based evaluators)
- Table stakes optimization levers: chunk size tuning, reranking, prompt engineering, and few shot examples
- Why the choice of LLM and embedding model both have an outsize impact on performance, and how to benchmark them against your own data
- Advanced techniques like small to big retrieval, embedding at the sentence level then expanding context before sending it to the LLM, and using agents as a reasoning layer on top of RAG
Start owning your AI