You will understand what RAG evaluation is, why it requires measuring both retrieval and generation separately, and how to instrument a Spring Boot AI system so you can tell — with numbers — whether your changes are making things better or worse.
Picture this: you've just deployed the EngineerPrep AI tutor. It answers learner questions by pulling relevant lesson chunks from pgvector and sending them to Claude on Amazon Bedrock. For the first few days, responses look great in your manual spot-checks. You demo it to the team. Everyone's impressed. Then the Slack messages start. A learner says the tutor confidently told them ConcurrentHashMap is just a slower HashMap . Another says it answered a Spring Boot 3 question by citing Spring Boot 1.5 docs. A third says the answer was technically correct but completely missed what the retrieved context was saying. You look at the logs for each one. The retriever did pull relevant chunks. Claude did receive them. Something in the chain broke — but where? The retriever? The prompt? The model? You have no dashboard, no metric, no score. You're debugging blind.…
RAG stands for Retrieval-Augmented Generation — a pattern where the AI looks up relevant documents first, then uses them to write its answer. Think of it like an open-book exam: the model gets to consult a reference before answering. RAG evaluation is simply the practice of scoring how well that open-book exam went. Here's the key insight most engineers miss: a RAG system has two separate jobs , and each can fail independently. Job 1 — Retrieval. Did the system find the right pages to look at? If it pulls the wrong context, even a capable model will give a bad answer. Job 2 — Generation. Given the right pages, did the model actually use them to write a good answer? A model can ignore its context, add hallucinated facts, or answer a different question entirely. Think of it like a research assistant. You could have a poor researcher who finds the wrong articles (retrieval failure).…
Step 1 — The Question Arrives A learner types: 'What is the difference between HashMap and ConcurrentHashMap?' Imagine this as a small envelope dropped into the system. It contains one thing: the question. --- Step 2 — Retrieval Happens The envelope travels to the pgvector search layer. The system turns the question into a vector (a list of numbers that captures its meaning) and searches the lesson database for the closest matches. Three chunks come back: - Chunk A: about HashMap internals - Chunk B: about thread safety in Java collections - Chunk C: about Iterator behavior in for-each loops Picture these as three sticky notes clipped to the envelope. Now ask: were these the right sticky notes? Chunks A and B look great. Chunk C is only loosely related. This is where Context Relevance is measured…