What is retrieval-augmented generation?

Retrieval-augmented generation, known as RAG, is a technique in which relevant documents are retrieved from a knowledge source and provided to a model as context, so that the answer is grounded in that data rather than relying only on what the model learned during training.

Does RAG retrain the model?

No. RAG does not change the model. It supplies relevant information at the time of the question by including retrieved text in the prompt. This allows current or private information to be used without the cost and effort of retraining.

What is an embedding?

An embedding is a list of numbers that represents the meaning of a piece of text. Text with similar meaning produces embeddings that are close together, which allows relevant documents to be found by comparing embeddings rather than by matching exact words.

Does RAG eliminate hallucination?

It reduces hallucination but does not remove it entirely. A hallucination is a confident but incorrect statement produced by a model. By grounding the answer in retrieved facts, RAG makes such statements less likely, although the quality of the retrieved data still matters.

How is RAG built on AWS?

Amazon Bedrock provides Knowledge Bases, which manage the retrieval step, the storage of embeddings, and the grounding of answers. The documents are typically stored in Amazon S3, and a vector store holds the embeddings.

What Is RAG (Retrieval-Augmented Generation)?

Introduction

Retrieval-augmented generation, commonly written as RAG, is a technique in which a model's answer is grounded in information that is retrieved from a knowledge source. A hallucination is a confident but incorrect statement produced by a model, and grounding the answer in retrieved facts makes such statements less likely. This guide explains what RAG is, why it exists, how it operates, and when it is appropriate. A general overview of the models involved is provided in the guide on Amazon Bedrock.

What is it?

RAG combines two steps. In the retrieval step, the documents that are most relevant to a question are found in a knowledge source. In the generation step, those documents are provided to the model as context, and the model produces an answer that is based on them. The word context refers to the text that is supplied to the model along with the question. Because the answer is grounded in supplied facts, it can reflect information that the model did not learn during training.

Why does it exist?

A foundation model is trained on a fixed set of data up to a certain point in time. It therefore cannot know information that is more recent, and it cannot know private information that was never part of its training. When asked about such matters, a model may produce a plausible but incorrect answer. RAG was developed so that current and private information can be supplied at the moment of the question, which improves accuracy without the model being retrained.

How it works

The documents in the knowledge source are first divided into small pieces called chunks, and each chunk is converted into an embedding, which is a list of numbers that represents its meaning. The embeddings are stored in a vector database, which is a store designed to find items by similarity of meaning. When a question is asked, it is also converted into an embedding, and the vector database is searched for the chunks whose embeddings are closest, a process known as semantic search. The retrieved chunks are added to the prompt, and the model generates an answer that is grounded in them.

Architecture diagram

Documents -> split into chunks -> embeddings -> vector database Question | convert to an embedding v Vector database --> returns the most similar chunks | v Prompt = question + retrieved chunks | v Foundation model --> generates an answer grounded in the chunks

Advantages

Current and private knowledge. Information that is not in the model can be used.
Reduced hallucination. Grounding the answer in facts makes incorrect statements less likely.
No retraining. The knowledge source can be updated without the model being changed.
Traceability. The retrieved sources can be shown to support the answer.

Disadvantages

Dependence on retrieval. If the retrieved documents are poor, the answer will be poor.
Added latency. The retrieval step adds time before the answer is produced.
Complexity. Chunking, embedding, and a vector store must be managed.
Cost. Storing embeddings and processing additional context adds to the cost.

Common use cases

Answering questions about a set of internal documents.
Customer support that draws on product manuals and policies.
Search over a large body of knowledge in natural language.
Assistants that must reflect current or private information.

Best practices

Documents should be divided into chunks of a sensible size, because chunks that are too large or too small reduce retrieval quality.
A suitable embedding model should be used so that meaning is captured accurately.
Only the most relevant chunks should be included, rather than as much text as possible.
The sources of an answer should be recorded so that it can be verified.

Common mistakes

Documents are chunked poorly, which causes irrelevant results to be retrieved.
Too much text is placed in the context, which raises cost and can reduce quality.
The quality of the retrieval is not evaluated, so poor answers are not detected.
RAG is expected to remove hallucination entirely, when it only reduces it.

Amazon Bedrock Knowledge Bases manage retrieval, embeddings, and grounding.
Amazon S3 stores the documents that form the knowledge source.
A vector store holds the embeddings that are searched.

Frequently Asked Questions

What is retrieval-augmented generation?: RAG is a technique in which relevant documents are retrieved from a knowledge source and provided to a model as context, so that the answer is grounded in that data rather than relying only on what the model learned during training.
Does RAG retrain the model?: No. RAG does not change the model. It supplies relevant information at the time of the question by including retrieved text in the prompt, which allows current or private information to be used without retraining.
What is an embedding?: An embedding is a list of numbers that represents the meaning of a piece of text. Text with similar meaning produces embeddings that are close together, which allows relevant documents to be found by comparing embeddings.
Does RAG eliminate hallucination?: It reduces hallucination but does not remove it entirely. By grounding the answer in retrieved facts, incorrect statements become less likely, although the quality of the retrieved data still matters.
How is RAG built on AWS?: Amazon Bedrock provides Knowledge Bases, which manage retrieval, the storage of embeddings, and grounding. The documents are typically stored in Amazon S3, and a vector store holds the embeddings.

Go deeper · Book as a Service™

Building AI Agents on AWS

This article is the summary. The book is the full, continuously updated reference: RAG with Knowledge Bases, embeddings, vector stores, and grounded agent design on AWS.

View the book