You already use RAG most weeks. NotebookLM, Claude Projects, a custom GPT with files attached: all the same idea. Here is what is happening under the hood, and why your answers get worse when you ignore it.
RAG turns a closed-book exam into an open-book exam: the AI still writes the answer from its own head, but now a few pages of your material sit in front of it. The catch is that a search step picks those pages, and if it grabs the wrong chunk, the model answers confidently from the wrong chunk. OpenAI’s file search splits documents into 800-token pieces by default[4], and a Stanford study of RAG-based legal tools found they still hallucinated 17% to 33% of the time[11]. This guide covers how retrieval works, why it fails, when RAG beats fine-tuning or pasting text in, and how to write documents that retrieve well.
RAG stands for retrieval-augmented generation, and the name is unusually honest. Something retrieves relevant text. That text is added to your prompt. Then the model generates an answer.
The mental model that works for almost everyone I teach: closed-book exam versus open-book exam.
A model answering from training alone is sitting the closed-book version. It has read enormously, it remembers imperfectly, and when it doesn’t know something it writes a fluent paragraph anyway, because fluent paragraphs are the whole job. With RAG, you hand it the textbook, open at the right page, seconds before it starts writing. Same student, completely different exam conditions.
The term comes from a 2020 NeurIPS paper by Patrick Lewis and eleven colleagues[1]. They pair a model’s “parametric memory,” knowledge baked into the weights during training, with “non-parametric memory,” in their case a searchable index of Wikipedia. They also named the two problems this was built to fix: models can’t tell you where an answer came from, and you can’t update what they know without retraining. Six years on, those are still the only two reasons anyone builds a RAG system.
My honest take on the term itself: “RAG” got dragged out of a research paper into a thousand vendor slide decks and stopped meaning anything specific in a sales conversation. When a platform says it “uses RAG,” you’ve learned almost nothing, because that describes NotebookLM, a seven-figure enterprise deployment, and a weekend project equally well. What matters is how good the retrieval is, and nobody puts that on the slide.
I run an exercise in workshops. Hands up if you’ve uploaded a document to an AI tool this month. Nearly every hand. Hands up if you know what RAG stands for. Two hands, maybe three, usually from someone in IT who came along for moral support.
Across roughly 2,000 learners that gap has been stable. Fine, right up until someone gets a wrong answer from their handbook and concludes the tool is broken.
NotebookLM. Google’s help documentation is explicit: “NotebookLM retrieves the most relevant information based on your question, then builds a response from it. A specific question helps NotebookLM locate the most relevant pieces of information.”[9] That’s RAG, described plainly for consumers, and it contains the most useful tip in this whole article. Specific questions retrieve better.
Claude Projects. Anthropic is unusually transparent about the switchover. A project starts by loading everything straight into the context window, originally 200,000 tokens, described at launch as “the equivalent of a 500-page book”[7]. Past that point: “When your project knowledge approaches the context window limit, Claude will automatically enable RAG mode to expand your project’s capacity by up to 10x.”[5] Claude then searches rather than reading everything at once[5], with no setup and no announcement[6]. Same files, quietly different mechanism, and most people never notice.
Custom GPTs with knowledge files. Up to 20 files, each up to 512 MB[3]. OpenAI spells out what happens next: “the GPT breaks the text up into chunks, creates embeddings (a mathematical way of representing text), and stores them for later use.”[3] Our custom GPT walkthrough covers the setup side.
And your company’s new internal chatbot, the one answering from the employee handbook: almost certainly RAG, and almost certainly nobody cleaned the document library first.
Every RAG system does roughly the same five things. Anthropic’s engineering write-up gives the standard version: break documents into chunks of “usually no more than a few hundred tokens,” turn each into a vector embedding, store those in a searchable database, then find the closest ones at question time[8].
The standard retrieval-then-generation sequence, as described in Anthropic’s engineering documentation on contextual retrieval [8] and OpenAI’s file search documentation [4]. This is a process diagram, not measured data.
Notice what the model is responsible for: step five only. Steps one to four are search engineering, happening before the clever part starts. If step three hands over the wrong three paragraphs, the model will write beautifully about the wrong paragraphs.
One detail from OpenAI’s documentation explains behaviour you’ve probably noticed: the file search tool rewrites your question before searching, can split a complex question into parallel searches, and runs keyword search alongside semantic search before reranking[4]. Semantic search alone misses exact strings. Anthropic’s example: search “Error code TS-999” and meaning-based search returns general content about error codes while missing the exact match[8]. That’s why quoting a phrase from your own document often beats describing it. You’re feeding the keyword half.
This is the claim most likely to be oversold to you in a procurement meeting, so I want to be careful with it. RAG genuinely helps: the original paper found RAG models “generate more specific, diverse and factual language” than the same model working from memory alone[1]. If you’ve read our explainer on why AI hallucinates in the first place, you’ll recognise why. You’re swapping a recall problem for a reading problem, and models read better than they recall.
But several legal technology vendors marketed their RAG products as “hallucination-free.” A Stanford team tested that in the first preregistered evaluation of these tools, published in the Journal of Empirical Legal Studies: the AI research tools from LexisNexis and Thomson Reuters “each hallucinate between 17% and 33% of the time.”[11] Lexis+ AI answered 65% of queries accurately; Westlaw’s tool, 42%[11]. Their summary line is the one I quote in workshops: “RAG systems are no panacea.”[11]
Read that again with your own handbook in mind. Those are expensive systems over curated legal databases. Your internal chatbot pointed at SharePoint will not do better.
OpenAI’s accuracy guide is blunt: “You can supply the wrong context, so the model can’t possibly answer, or you can supply too much irrelevant context, which drowns out the real information and causes hallucinations.”[2] The third failure is that the model gets the right context and does the wrong thing with it[2]. A team at Deakin University catalogued seven distinct failure points across three real deployments, most of which belong to you and your document library rather than the model[12].
Their most uncomfortable conclusion, and the one I’d put on a procurement checklist: “validation of a RAG system is only feasible during operation.”[12] You cannot tell whether it works from a demo. You find out in month three.
Here is the part almost nobody explains to business users, and the part that would fix most of their complaints.
Your documents don’t get stored whole. They get cut up. OpenAI’s file search splits every file into 800-token chunks with 400 tokens of overlap by default[4], roughly 600 words. Gemini’s File Search lets you set your own tokens per chunk and overlap[10]. So the unit of retrieval is a page. Not your document. A page.
Anthropic’s engineering team gives the cleanest illustration. A chunk reads: “The company’s revenue grew by 3% over the previous quarter.” Perfectly good sentence. Now retrieve it when someone asks about ACME Corp’s Q2 2023 revenue growth. The chunk doesn’t say ACME, or Q2, or 2023. Cut from its document, it’s nearly unfindable[8]. Their fix: have a model write one sentence describing where each chunk came from and prepend it before indexing. Contextual embeddings cut the top-20 retrieval failure rate by 35%, adding a keyword index took it to 49%, and reranking to 67%, moving failures from 5.7% of queries to 1.9%[8].
Before anything can be chunked it has to be parsed out of your file, and a surprising share of “the AI is wrong” complaints start right here. OpenAI says it plainly: “A single column of text is best. The parser can struggle with multi-column PDFs, and won’t understand the nuance conveyed by the relative positions of text on a PowerPoint slide.”[3] NotebookLM refuses a source outright if the PDF is copy-protected[9].
Now think about where corporate knowledge lives. Two-column PDF policies. Slides where the meaning is in a diagram. Scanned contracts. The parser will partly or wholly fail on each, nothing tells you, and the chatbot answers as though that policy doesn’t exist.
One more gap, from OpenAI’s own list: “Better support for summarization: the tool today is optimized for search queries.”[4] Ask a RAG system to summarise a whole document and you’re asking a search tool to do a reading tool’s job. Our guide on summarising long documents covers what to do instead.
Somebody in your organisation is going to propose fine-tuning a model on your company data. Learn this distinction first, because it will save you a six-figure decision. Fine-tuning teaches a model how to behave; RAG gives it something to read.
OpenAI lists exactly two reasons to fine-tune: to improve accuracy on a task by showing the model many correct examples, and to get the same accuracy from fewer tokens[2]. Neither is “so it knows our products.” If the model doesn’t know your Q3 pricing, fine-tuning is the wrong tool, and it’ll still be wrong when the pricing changes in October.
The underrated option is the third one: paste the thing in. Anthropic puts a number on it. “If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt.”[8] Five hundred pages covers most real business use cases.
| Approach | Fixes | Use when | Watch out for |
|---|---|---|---|
| Paste into the prompt | The model hasn’t seen this document | Your material fits in the context window (Anthropic suggests under 200k tokens, roughly 500 pages) [8] | Models retrieve less reliably from the middle of long inputs [13] |
| RAG | The knowledge base is too big to paste | Content changes often, you need citations, or the library is large [1] | Retrieval becomes your bottleneck, and it fails silently [12] |
| Fine-tuning | Wrong format, wrong tone, wrong reasoning steps | You need consistent behaviour, or the same accuracy from fewer tokens [2] | It does not reliably add facts, and goes stale when your facts change [2] |
Decision summary drawn from OpenAI’s LLM accuracy guidance [2] and Anthropic’s contextual retrieval post [8]. A framework for choosing, not benchmark data.
One real caveat on the paste option. A Stanford-led study in Transactions of the ACL found that “performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts.”[13] Practical translation: paste twelve documents with the answer buried in document seven and your odds drop. Paste four, and put the one you care about first or last.
These techniques do “stack on top of each other”[2]. Just don’t start there.
This is the highest-value thing a non-technical person can do, and it needs no permissions from IT. Every rule below follows from one fact: each chunk has to make sense alone.
Repeat the nouns. A paragraph starting “It applies to all employees hired after that date” is dead on arrival as a chunk. “This parental leave policy applies to all employees hired after 1 January 2026” survives being cut out of its document. Clunkier for a human reader, who has the whole page and will cope.
Put the answer next to the heading. Chunk boundaries fall near structural breaks. Three paragraphs of throat-clearing between a heading and the actual policy is how you end up with a chunk containing only throat-clearing.
Name files by their contents. Anthropic’s own advice for RAG-enabled projects: “Well-named files help Claude understand and retrieve the right information more effectively.”[5] “Policy_v4_FINAL_updated.docx” tells the system nothing.
Date-stamp anything that expires. When two chunks contradict each other the retriever has no idea which is current, so it hands both over and the model picks or blends. Put “Effective 1 January 2026, replaces the 2024 version” in the body text, not a footer the parser may drop.
Kill the duplicates and convert the awkward formats. The most common cause of a confidently wrong internal chatbot answer isn’t a model failure. It’s three versions of one policy in the same folder, two superseded, all three indexed. And two-column PDFs need flattening to single-column[3].
And one habit that isn’t about writing at all: ask for sources, then click through. Custom GPTs avoid naming uploaded files unless instructed otherwise[3], and NotebookLM skips individual citations when a source is short[9]. Without checking, you can’t tell whether an answer came from your policy or the model’s general knowledge of policies. Our anti-hallucination toolkit goes further on verification.
At Future Factors we frame every rollout the same way, and it applies to RAG better than to almost anything else we teach:
Tool x Workflows x Behavior = AI-powered professional.
Multiplication, deliberately, not addition. A zero anywhere zeroes the whole thing, and you can’t compensate by doubling another factor. Which is exactly what organisations attempt when they buy a better tool to fix adoption.
Tool is the licence: NotebookLM, a Claude Project, a custom GPT. Bought first, matters least on its own.
Workflows is the document library the tool retrieves from and the process keeping it current. For RAG this factor does most of the work. If your handbook is a two-column scanned PDF and the folder holds three versions of it, your workflow factor is near zero and no licence upgrade moves it.
Behavior is what the person does: asking a specific question instead of a vague one, checking the citation, speaking up when an answer contradicts what they know.
I’ve watched teams with mediocre tools and clean documents beat teams with the newest platform and a folder of scanned PDFs. Every time.
And I used to teach RAG as a technical concept, with the vector database and the embeddings and a diagram. People nodded and forgot it. The version that sticks is the exam metaphor plus one instruction: make the open book easy to read. That turns an IT problem into a writing problem, and writing is something everyone already knows how to do.
So take one action from all this. Open the folder your AI tool points at, find the three documents people ask about most, rewrite the headings so each section stands alone, spell out the acronyms, delete the superseded copies, re-upload. Then ask your five most common questions again. That’s an afternoon, and it will do more for your answer quality than any model upgrade shipping this year.
RAG stands for retrieval-augmented generation. Before the AI writes an answer, a search step pulls relevant passages from your documents and adds them to the prompt, so the model reads rather than recalls. The term comes from a 2020 NeurIPS paper that paired a model’s “parametric memory” with a searchable index its authors called “non-parametric memory”[1]. Answering from training alone is a closed-book exam; RAG hands it the open book.
It reduces hallucination and does not remove it. Several legal technology vendors advertised RAG products as hallucination-free, and a preregistered Stanford study in the Journal of Empirical Legal Studies found the leading tools still hallucinated between 17% and 33% of the time[11]. Their phrase: RAG systems are “no panacea.” Failures come from retrieving the wrong passage, retrieving so much noise it drowns the right answer, or the model misreading a correct passage[2].
Your documents are not stored whole. They are cut into pieces and indexed separately, so each piece has to make sense alone to be findable. OpenAI’s file search uses 800-token chunks with 400 tokens of overlap, and passes at most 20 chunks into the prompt[4]. Anthropic’s example: a chunk reading “The company’s revenue grew by 3% over the previous quarter” can’t be retrieved by a question naming the company and quarter, because it mentions neither[8].
Use RAG when the model needs information it doesn’t have: your documents, your policies, anything that changes. Use fine-tuning when it has the information but behaves wrongly. OpenAI lists exactly two reasons to fine-tune: improving accuracy on a task through examples, and getting the same accuracy from fewer tokens[2]. Neither is “so it knows our products.” Check the third option first: under roughly 200,000 tokens, about 500 pages, Anthropic suggests putting it all in the prompt and skipping RAG[8].
Clean the source folder first, deleting superseded versions so contradictory chunks stop competing. Convert awkward files, since OpenAI’s parser struggles with multi-column PDFs and can’t read a slide’s layout[3], and NotebookLM rejects copy-protected PDFs outright[9]. Name files descriptively, which Anthropic recommends for RAG-enabled projects[5]. Then ask narrow questions and quote an exact phrase when you know one[8].
This explainer is based on Future Factors’ work teaching document-grounded AI tools to non-technical teams, checked against primary sources: the original Lewis et al. RAG paper and the current published documentation from OpenAI, Anthropic and Google. Widely-quoted figures claiming RAG cuts hallucination by a specific percentage traced back only to vendor blogs citing other vendor blogs, so they were excluded and the omission is stated in the text. All documentation was verified on 21 August 2026.
Weekly tips. Real examples. Practical help for busy professionals.
We care about your data, check out our privacy policy.