By the end of this lesson you will understand how Retrieval-Augmented Generation works from embedding generation through vector search to grounded LLM response — and why every architectural decision in that pipeline exists to solve a specific production failure mode.
It's a Tuesday afternoon. A user pastes an 8,000-word technical document into EngineerPrep's AI tutor and asks: "Can you summarize the key interview topics from this?" The response from the backend? A 400 error. com.engineerprep.llm.LlmClientException: Token limit exceeded. Model: claude-3-sonnet. Max tokens: 4096. Received: 11,247. 8,000 words. That's not a book. That's a long blog post. And the model can't handle it. You increase the context window. Now it's expensive — Claude Opus at 100k tokens costs roughly $15 per million input tokens. A single user session with one large document is burning $0.15. Multiply by a thousand active users, and your AWS bill just became a line-item conversation in a quarterly review. You try a different angle: summarize the document before sending it. But now the AI tutor is answering based on a summary of the original content…
We just watched the AI tutor fail because it was trying to hold everything in its head at once. RAG is the fix. What exactly is RAG? Retrieval-Augmented Generation is an architecture where, before an LLM generates a response, a retrieval system finds the most relevant pieces of your private knowledge base and injects them directly into the prompt. The model then generates its answer grounded in those retrieved facts — not from its training data alone. Think of it like the difference between asking a surgeon a question cold versus handing them the patient's chart first. The surgeon is equally skilled either way, but with the chart they're answering about this patient , not the average patient. Why was it invented? LLMs are trained on a snapshot of the world. Claude, GPT-4, and every model you'll use in production stopped learning at a cutoff date.…
Scene Setup Imagine two rooms separated by a hallway. On the left is a library — floor-to-ceiling shelves, each shelf holding a different topic. Java concurrency on one shelf. Spring Boot configuration on another. System design patterns on a third. Each book on each shelf has been read, indexed, and given a numeric fingerprint — a vector — that captures its meaning. This is your vector store (pgvector in EngineerPrep's case). On the right is a very smart analyst — Claude. The analyst is brilliant but has no access to the library on their own. They only know what you put on their desk. In the hallway stands a retrieval agent — a Spring Boot service. Its job is to run between rooms. Animation: The Query Arrives A user types: "How does Spring handle circular dependencies?" Watch that sentence travel into the hallway. The retrieval agent catches it.…
The Two Pipelines RAG has two entirely separate execution paths that engineers frequently conflate, and conflating them causes production mistakes. Ingestion pipeline (offline, batch): runs when content changes. Takes raw documents, splits them into chunks, embeds each chunk, stores (chunk text, vector, metadata) in PostgreSQL. Query pipeline (online, per-request): runs for every user question. Embeds the query, searches for similar vectors, assembles the prompt, calls the LLM. These pipelines have completely different performance characteristics. Ingestion is throughput-bound and can be async. The query pipeline is latency-bound and is on the critical path of a user request. --- Embeddings: What the Numbers Actually Are An embedding model (Bedrock Titan Text Embeddings v2 in EngineerPrep's case) maps text to a point in ℝ^1536 — a 1,536-dimensional vector space.…