- RAG combines a retriever (usually a vector database or knowledge graph) with a generator (an LLM).
- It reduces hallucinations, enables source citation, and lets you use current data without retraining.
- A RAG pipeline runs in three phases: indexing, retrieval, and generation.
- Common variants: standard RAG, hybrid RAG, Graph RAG, and Agentic RAG.
- Enterprise h2oGPTe includes an end-to-end RAG pipeline, and H2O Eval Studio scores its retrieval, accuracy, and groundedness.
What is RAG?
RAG turns an LLM from a "closed-book" generator that relies only on its training data into an "open-book" system that can cite current, proprietary, and domain-specific information.
Retrieval-Augmented Generation was introduced in a 2020 Facebook AI Research paper (Lewis et al.) as a way to combine a pre-trained sequence-to-sequence model with a dense retriever. The core insight: instead of trying to bake every fact into the model's weights, keep the facts in a searchable index and hand the relevant ones to the model at query time.
A RAG system solves two persistent LLM problems at once. First, an LLM's knowledge is frozen at training time; RAG lets you serve information created yesterday. Second, an LLM confabulates when it doesn't know; RAG gives it real passages to summarize instead of forcing it to guess. RAG is the default architecture for enterprise GenAI in 2026 for exactly these reasons.
RAG is closely tied to embeddings (the technique that turns text into vectors for similarity search), tokenization (how the model reads text), the context window (how much retrieved text you can pass in), and prompt engineering (how retrieved context is presented to the model).
How does RAG work?
A production RAG pipeline runs in three phases: indexing, retrieval, and generation.
- Indexing (offline). Documents are chunked, each chunk is turned into an embedding vector, and the vectors are stored in a vector database. Metadata (source, section, permissions) is stored alongside so results can be filtered and cited. See the Enterprise h2oGPTe ingestion docs for supported formats and chunking options.
- Retrieval (per query). The user query is embedded and compared to the index. The top-k most similar chunks are returned. Modern pipelines combine keyword search and vector search (hybrid retrieval) and often add a reranker to reorder results, a step semantic search systems rely on.
- Generation (per query). The retrieved chunks are formatted into the LLM's prompt as context, along with the user question and a system prompt that tells the model to answer from the passages and cite them. The LLM writes the answer, and H2O Eval Studio can score the result for groundedness.
RAG variants
Variant
What it does
When to use
Standard RAG
One retrieval, one generation
Q&A, document search
Hybrid RAG
Combines keyword and vector search, plus reranking
Production systems needing high recall
Retrieves from a knowledge graph, not just documents
Entity-rich data, complex relationships
An agent decides when, what, and how many times to retrieve
Multi-hop reasoning, dynamic workflows
Benefits of RAG
Fewer hallucinations. Grounding the answer in retrieved passages cuts fabricated claims.
Fresh, private data without retraining. New documents show up as soon as they're indexed. No fine-tuning cycle, no full retraining. Data stays inside your control boundary.
Source citations. Because RAG retrieves specific chunks, the system can cite them. Regulated industries (banking, healthcare, government) treat this as table stakes.
Smaller models can win. A well-built RAG pipeline lets a smaller, cheaper model outperform a bigger model that has to guess, cutting inference cost significantly.
Challenges of RAG
Retrieval quality dominates. Your generation is only as good as your retrieval. Chunking strategy, embedding model choice, hybrid vs. pure vector, and reranking all matter, often more than the LLM.
Latency budget. Adding a retrieval step adds latency. Real-time apps need index tuning, caching, and reranker discipline.
Context window limits and cost. Stuffing too much retrieved text into the prompt increases cost and can reduce answer quality ("lost in the middle"). Retrieve less, more precisely.
Evaluation is its own problem. You need to score retrieval (did we find the right chunks?), generation (is the answer accurate?), and groundedness (is the answer supported by the retrieved text?). See RAG evaluation.
RAG use cases
- Enterprise knowledge management: employees query internal wikis, policies, and runbooks in natural language.
- Customer support: assistants retrieve product docs to answer product questions with citations.
- Legal and compliance research: search case law, regulations, and internal opinions with source attribution.
- Clinical decision support: clinicians query guidelines and literature at the point of care.
- Financial analysis: analysts query filings, earnings calls, and market data.
RAG in H2O AI
Enterprise h2oGPTe provides an end-to-end RAG pipeline (ingestion, chunking, embedding, retrieval, generation) over documents such as PDF, DOCX, and PPTX, plus structured data, with Graph RAG and Agentic RAG built in. H2O Eval Studio scores retrieval quality, generation accuracy, and groundedness. Both run on-prem and air-gapped, so RAG works on data that never leaves your control.