Skip to main content

How RAG Actually Works (Without the Hype)

Ask a raw language model "What did Ahmed work on in London?" and you get one of two bad answers: an honest "I don't know," or — worse — a confident, fluent, completely invented one. RAG is how you fix that. Not by retraining the model. By handing it the right page of the book at the exact moment it needs it.

The problem RAG solves​

A trained LLM is frozen. Its knowledge stops at a cutoff date, and it has never seen your private documents — your CV, your codebase, your research notes. It's brilliant at language and unreliable at facts it was never shown.

Without RAG you have two expensive options:

  • Fine-tune the model on your data → costly, slow, and stale the moment your data changes.
  • Paste everything into the prompt → you hit context limits, pay for every token on every call, and bury the relevant fact under noise.

Retrieval-Augmented Generation (RAG) is the third path: keep the model frozen, and at question time, fetch only the relevant facts and place them next to the question.

The open-book exam analogy. The student (the LLM) is smart but hasn't memorized your textbook. RAG is the librarian who, the instant you ask a question, slides exactly the right two pages across the desk. The student doesn't get smarter — they get informed.

The pipeline in five moves​

RAG has two phases: indexing (done once, ahead of time) and retrieval + generation (done per question).

Indexing — offline, build it once:

  1. Chunk — split your documents into passages of roughly 200–500 tokens. Too large and each chunk is noisy; too small and you lose the surrounding context.
  2. Embed — convert each chunk into a vector (a list of numbers) using an embedding model. Similar meaning → nearby vectors. This is the core trick: meaning becomes geometry.
  3. Store — load those vectors into a vector database (I use Pinecone) that can find nearest neighbors in milliseconds.

Query time — online, runs per question:

  1. Retrieve — embed the user's question with the same model, then pull the top k closest chunks by cosine similarity.
  2. Generate — drop those chunks + the original question into a prompt and send it to the LLM (I run open-source Llama models on Groq for fast inference). The model answers grounded in the retrieved text.
# The whole idea in ~8 lines
question = "What did Ahmed do in London?"
q_vector = embed(question)                      # same model used at indexing
chunks   = vector_db.query(q_vector, top_k=4)   # nearest passages by meaning
prompt   = f"Context:\n{chunks}\n\nQuestion: {question}\nAnswer using only the context."
answer   = llm.generate(prompt)                 # grounded, not guessed

Rag Photo

Why embeddings are the part that matters​

A little deeper, for the curious: an embedding maps text into a high-dimensional space — say 768 or 1024 dimensions — where distance approximates semantic difference. The phrases "APT detection" and "catching advanced persistent threats" land close together even though they share almost no words.

That's the whole reason RAG beats plain keyword search: it matches meaning, not strings. Get the embedding model right and retrieval quality follows; everything downstream depends on it.

Where RAG quietly fails (the honest part)​

RAG is a retrieval system, not a truth machine. It breaks in predictable ways:

  • Bad chunking. Split a paragraph mid-thought and retrieval hands the model half an idea.
  • Retrieval misses. If the answer isn't in the top-k chunks, the model can't use it — and may hallucinate to fill the gap.
  • Garbage in, garbage out. RAG faithfully retrieves from a messy knowledge base. Clean your data, or it will repeat your mistakes with total confidence.
  • No reasoning across documents. Vanilla RAG pulls independent chunks. If a fact lives in pieces spread across many sources, simple similarity search struggles to connect them.

None of this is a reason to avoid RAG. It's the reason to build it carefully — which is exactly what separates a demo from a system that works.

How this looks in a real build​

The digital twin on bargady.online started as exactly this pipeline: my CV, project write-ups, and blog notes → chunked → embedded → stored in Pinecone → retrieved per question → answered by an open-source LLM on Groq.

Ask it "What did Ahmed do in London?" and instead of guessing, it retrieves the chunk about the PayRue blockchain role and answers from it — grounded, and traceable back to the source.

Ahmed

The one-sentence takeaway​

RAG doesn't make the model smarter — it makes the model informed, by retrieving the right context at the right time.

Part of the build-in-public series on docs.bargady.online. I build real AI systems and explain how they actually work — no hype.