What Is RAG? A Practical Guide for Developers

This is what a language model sees in a RAG system:

Answer the question using only the sources below.
Cite sources by number, like [2]. If the sources don't contain the answer, say you don't know.
 
Sources:
[1] employee-handbook.pdf, section 7.2
Equipment over $500 needs written approval from your manager before purchase.
 
[2] remote-work-policy.md
Remote employees may expense one monitor per year, up to $300, with an itemised receipt.
 
Question: Can I expense a monitor for working from home?

The model has never seen this company's handbook. It doesn't need to. The application found the two relevant paragraphs, pasted them into the prompt, and asked the model to answer from them. The model's job shrinks from knowing the answer to reading it.

That is the whole idea. Retrieval-Augmented Generation (RAG) is an architecture in which an application retrieves relevant text from your own data at request time and adds it to the prompt, so a language model can answer from information it was never trained on. Everything else in this guide is about doing those two steps, retrieving and prompting, well enough to trust the answer.

What RAG solves, and what it doesn't

A language model only knows its training data, frozen at a cutoff date. It has never seen your documentation, your database schema, last week's support tickets, or your internal policies. RAG fixes exactly that gap:

• Private and recent knowledge. Re-index a document and the next answer uses it. No retraining.
• Citations. Because you know which chunks went into the prompt, you can show the user where an answer came from.
• Smaller blast radius for hallucination. "Answer only from these sources, otherwise say you don't know" is an instruction a model can actually follow when the sources are in front of it.

RAG does not make a model reason better, follow a new output format, or adopt a writing style. Those are behaviour problems, not knowledge problems. And it can't answer from text it fails to retrieve. Most RAG bugs are retrieval bugs.

How RAG works: two pipelines

A RAG system is two pipelines that share a vector store.

INGESTION (offline, whenever documents change)
 
  Documents ──► Extract text ──► Chunk ──► Embed ──► Vector store
                                                      (vector + text + metadata)
 
QUERY (online, every request)
 
  Question ──► Embed ──► Search vector store ──► Top-k chunks
                                                     │
                     Answer ◄── LLM ◄── Prompt = instructions + chunks + question

Ingestion turns documents into searchable pieces. Query finds the pieces relevant to a question and hands them to the model. The rest of this guide walks through each step with C# you can run against a local model.

Step 1: Chunk your documents

You can't put a whole manual in every prompt, and a single embedding of a 50-page PDF says almost nothing about any one page. So documents are split into chunks, typically a few paragraphs each, and each chunk is retrieved independently.

A paragraph-aware chunker with overlap is a sound default:

static List<string> Chunk(string text, int maxChars = 1500, int overlapChars = 200)
{
    var chunks = new List<string>();
    var current = new System.Text.StringBuilder();
 
    foreach (var paragraph in text.Split("\n\n", StringSplitOptions.RemoveEmptyEntries))
    {
        if (current.Length > 0 && current.Length + paragraph.Length > maxChars)
        {
            var chunk = current.ToString();
            chunks.Add(chunk);
            // Carry the end of the previous chunk forward so a fact split across
            // the boundary still appears whole in one of the two chunks.
            current.Clear().Append(chunk[Math.Max(0, chunk.Length - overlapChars)..]).Append("\n\n");
        }
        current.Append(paragraph.Trim()).Append("\n\n");
    }
 
    if (current.Length > 0) chunks.Add(current.ToString().Trim());
    return chunks;
}

The chunking decisions that matter in practice:

• Split on structure, not on character counts alone. Paragraphs, headings, and list items are natural boundaries. Cutting mid-sentence produces chunks that mean nothing on their own.
• Keep chunks small enough to be specific. A chunk that covers five topics matches every question weakly and none strongly. A few hundred words is a common starting point; tune it against your own questions (see the evaluation section below).
• Store metadata with every chunk: source file, section heading, page, last-modified date, and who is allowed to read it. You'll need it for citations, filtering, and permissions.
• Prepend context the chunk lost. "It must be approved in writing" is useless without knowing it came from Equipment purchases. Adding the document title and heading to the chunk text before embedding is cheap and helps retrieval.

Step 2: Turn chunks into embeddings

An embedding model converts text into a vector, a list of numbers (768 of them for nomic-embed-text), positioned so that texts with similar meaning end up close together. "Can I buy a monitor?" and "equipment purchase approval" share almost no words but land near each other.

With Ollama running locally, embedding a batch of chunks is one HTTP call:

using System.Net.Http.Json;
 
record EmbedRequest(string Model, string[] Input);
record EmbedResponse(float[][] Embeddings);
 
var ollama = new HttpClient { BaseAddress = new Uri("http://localhost:11434") };
 
async Task<float[][]> EmbedAsync(string[] texts)
{
    var response = await ollama.PostAsJsonAsync("/api/embed", new EmbedRequest("nomic-embed-text", texts));
    response.EnsureSuccessStatusCode();
    return (await response.Content.ReadFromJsonAsync<EmbedResponse>())!.Embeddings;
}

Two rules prevent most embedding bugs:

1. Use the same model for documents and questions. Vectors from different models live in different spaces; comparing them returns noise, not an error.
2. Re-embed everything when you change models. Plan for it from the start: store the model name with the vectors.

Some models also expect a task prefix. nomic-embed-text, for example, is trained with search_document: in front of indexed text and search_query: in front of questions. Check your model's documentation; skipping the prefix quietly lowers retrieval quality.

Step 3: Store and search vectors

A vector store keeps each chunk's vector alongside its text and metadata, and answers one question fast: which stored vectors are closest to this one? Closeness is usually cosine similarity, the cosine of the angle between two vectors, from −1 (opposite) to 1 (same direction):

static float CosineSimilarity(float[] a, float[] b)
{
    float dot = 0, normA = 0, normB = 0;
    for (var i = 0; i < a.Length; i++)
    {
        dot += a[i] * b[i];
        normA += a[i] * a[i];
        normB += b[i] * b[i];
    }
    return dot / (MathF.Sqrt(normA) * MathF.Sqrt(normB));
}

(In production .NET code, TensorPrimitives.CosineSimilarity from System.Numerics.Tensors does this with SIMD.)

For a few thousand chunks, scanning them all in memory with that function is genuinely fine and a good way to learn. Beyond that, use a store with an approximate-nearest-neighbour index. Common choices:

The vector database is one component, not the system. Answer quality is decided more by chunking, retrieval, and prompting than by which store you pick.

Step 4: Retrieve the right chunks

At query time you embed the question, search for the top k chunks (often 3–8), and pass them on. This is the step that fails most often, and it fails silently: the model still produces a fluent answer from the wrong context.

We measured this in the AI Database Agent, which retrieves business definitions from a 40-entry glossary in Qdrant using nomic-embed-text:

• Dense search alone found only 3 of 6 relevant definitions in the top five. Short texts produced compressed similarity scores (0.52–0.77), so "How much do our warehouse staff earn?" never reached the note about salaries.
• Two standard fixes changed that. Each entry got "also known as" terms (the words people actually use: "AOV", "earn", "YTD"), and retrieval became hybrid: an exact match on a term or alias goes in first, and dense search fills the remaining slots for paraphrases.
• Recall on 12 paraphrases written after the fix, never used to tune it, was 10 of 12.
• Similarity thresholds need measuring, not guessing. True paraphrases of a worked example scored 0.77–0.88; related-but-different questions scored up to 0.77. A threshold of 0.60 let misleading examples into the prompt, so it was raised to 0.80.

Things worth doing in almost every RAG pipeline:

• Hybrid search. Combine vector similarity with keyword matching (BM25 or exact term match). Vectors handle paraphrase; keywords handle product codes, names, acronyms, and error messages, which embeddings routinely blur.
• Metadata filters. Restrict by product, version, language, or date before ranking, so a question about v3 never retrieves v2 documentation.
• Permission filtering at retrieval time. If a user can't open a document, its chunks must never reach their prompt. Filtering the answer afterwards is too late.
• Reranking (optional). Retrieve 20–50 candidates cheaply, then re-score them with a cross-encoder reranker and keep the best few.

Step 5: Build the prompt

Retrieved chunks are just text until you frame them. A prompt that numbers the sources, demands answers from them, and permits "I don't know" does most of the anti-hallucination work:

record Chunk(string Source, string Text);
 
static string BuildPrompt(string question, IReadOnlyList<Chunk> chunks)
{
    var sources = string.Join("\n\n", chunks.Select((c, i) => $"[{i + 1}] {c.Source}\n{c.Text}"));
 
    return $"""
        Answer the question using only the sources below.
        Cite sources by number, like [2]. If the sources don't contain the answer, say you don't know.
 
        Sources:
        {sources}
 
        Question: {question}
        """;
}

Keep the instructions short and put the most relevant chunk first or last; models attend less reliably to the middle of a long context. Number the sources so citations in the answer map back to real documents you can link.

Step 6: Generate the answer

Generation is the simplest step: send the prompt to a chat model.

record ChatMessage(string Role, string Content);
record ChatRequest(string Model, ChatMessage[] Messages, bool Stream);
record ChatResponse(ChatMessage Message);
 
async Task<string> AnswerAsync(string question)
{
    var questionVector = (await EmbedAsync([question]))[0];
    var top = index                                   // your stored (Chunk, float[] Vector) pairs
        .OrderByDescending(e => CosineSimilarity(e.Vector, questionVector))
        .Take(5)
        .Select(e => e.Chunk)
        .ToList();
 
    var response = await ollama.PostAsJsonAsync("/api/chat",
        new ChatRequest("llama3.1", [new("user", BuildPrompt(question, top))], Stream: false));
    response.EnsureSuccessStatusCode();
    return (await response.Content.ReadFromJsonAsync<ChatResponse>())!.Message.Content;
}

That's a complete, working RAG loop in under 100 lines of C#: chunk, embed, search, prompt, generate. The ASP.NET Core and Qdrant tutorial turns it into an API with a real vector database.

RAG vs fine-tuning vs long context

RAG is not the only way to give a model more knowledge. The three options solve different problems:

They combine well. A fine-tuned model can follow your answer format while RAG supplies the facts, and a long context window lets RAG pass more chunks. If your knowledge changes or must be cited, RAG is almost always part of the answer.

Where RAG breaks

Each of these failure modes produces a confident, well-written, wrong answer:

• Retrieval missed. The answer exists in your data but wasn't in the top k. Fix with hybrid search, better chunking, aliases, or more candidates plus reranking.
• Retrieved but not followed. In the AI Database Agent, the correct definition of average order value was retrieved by exact match, and the 7B model still averaged order lines. Retrieval puts facts in front of the model; it doesn't guarantee the model weighs them over its habits. Validate outputs where you can.
• Stale index. A policy changes, the old chunk stays. Re-index on change and store a lastModified date with every chunk.
• Bad chunk boundaries. A table split across two chunks, or a rule separated from its exception.
• Prompt injection through documents. Retrieved text is untrusted input. A document that says "ignore previous instructions" is now in your prompt. Treat retrieved content as data, never as instructions; see AI agent security.
• Permission leaks. Without access filtering at retrieval, RAG will cheerfully quote the salary spreadsheet to anyone who asks the right question.

How to evaluate a RAG system

"It looks right" is not an evaluation. Test retrieval and generation separately, because they fail for different reasons and need different fixes.

1. Build a golden set. 30–100 real questions, each with the document or chunk that answers it and a reference answer.
2. Measure retrieval first. For each question: is the right chunk in the top k (recall@k), and how high (reciprocal rank)? If retrieval recall is low, no prompt will save you.
3. Then measure answers. Is the answer correct, and is every claim supported by the retrieved sources (faithfulness)? An LLM can grade faithfulness at scale, but spot-check its grades by hand.
4. Keep a held-out set. Questions you never look at while tuning. In the AI Database Agent, the development set reached 100% after tuning; the number worth quoting was the held-out set, which rose from 50% to 75%. Tuning against your test questions measures memory, not quality.

Re-run the set on every change to chunking, embedding model, retrieval, or prompt. RAG quality regresses quietly.

A practical RAG checklist

• Chunks follow document structure, keep a few hundred words, and carry title and heading context.
• Every chunk stores source, section, date, embedding model, and access permissions.
• Documents and questions use the same embedding model, with the model's query/document prefixes.
• Retrieval is hybrid: vector similarity plus keyword or exact-term matching.
• Metadata and permission filters apply before ranking.
• Similarity thresholds are measured on real questions, not guessed.
• The prompt numbers sources, requires citations, and allows "I don't know".
• Retrieved text is treated as untrusted data.
• A golden set with a held-out portion runs on every change.

Frequently asked questions

What does RAG stand for?

RAG stands for Retrieval-Augmented Generation. The application retrieves relevant information from your data and uses it to augment the prompt before the language model generates an answer.

Do I need a vector database for RAG?

Not to start. For a few thousand chunks, an in-memory list and cosine similarity work fine. Use a vector database (Qdrant, pgvector, SQL Server's vector type) when you need scale, filtering, persistence, or hybrid search.

Is RAG better than fine-tuning?

They solve different problems. RAG adds knowledge that changes or must be cited; fine-tuning changes how a model behaves, such as its tone or output format. Many production systems use both.

Does RAG stop hallucinations?

It reduces them when retrieval finds the right context and the prompt requires answers from the sources. It can't prevent answers built on the wrong chunk, or a model ignoring the right one, which is why evaluation matters.

Can I build RAG in .NET?

Yes. Everything in this guide is C#: HttpClient calls to a local model through Ollama, an in-memory index, and a prompt builder. For a full API with a vector database, see Build RAG with ASP.NET Core and Qdrant.

Takeaways

1. RAG is retrieval plus a prompt. The model reads your data at request time instead of memorising it.
2. Retrieval quality decides answer quality. Invest in chunking, hybrid search, and filters before swapping models.
3. Measure, don't guess. Similarity thresholds, chunk sizes, and k are empirical. Our own glossary needed hybrid retrieval to go from 3 of 6 relevant definitions to 10 of 12.
4. Retrieved isn't followed. Validate outputs and keep a held-out evaluation set.
5. Treat retrieved text as untrusted and filter by permissions at retrieval time.

Next: build it for real with ASP.NET Core and Qdrant, using a model running locally with Ollama.