Evaluation is the practice of systematically checking whether your AI system gives correct, useful answers — and you will understand why it is a reliable way to catch the silent failures that unit tests miss.
You merge a pull request that swaps Claude 2 for Claude 3 on EngineerPrep's AI tutor. The change looks safe — same prompt, newer model, better benchmarks. A week later, a learner posts in support: 'The tutor told me Collections.synchronizedList makes a list fully thread-safe for iteration. I used that in my interview and got it wrong.' You grep the codebase. No bug. You run the unit tests. All green. You manually ask the tutor the same question. Today it answers correctly. So what happened? The model changed. The prompt didn't. Some questions got better; some got quietly worse. And you had no way to know — because you never measured. That measurement is called evaluation , or evals for short. This lesson explains what it is, why it exists, and how to build your first one.
Think about a chef tasting a new dish before it leaves the kitchen. They aren't guessing whether the food is good — they are checking against a known standard: the right seasoning, the right texture. If something is off, they fix it before the customer sees it. Evaluation is the same idea applied to AI. An eval is a small, repeatable test that sends a known question to your AI system and checks whether the answer meets a standard you defined in advance. Notice three parts: 1. A known question — an input you control. 2. A standard — what a correct or acceptable answer looks like. 3. A check — some code (or another AI) that compares the answer to the standard. That's it. No magic. The reason evals exist is that AI output is not deterministic the way normal code is. When you call list.get(0) you know exactly what you'll get.…
Imagine a simple timeline moving left to right. Here is what happens during one eval run. Step 1 — The question enters. A box labeled EvalRunner holds a list of test cases. Each test case is a pair: a question and an expected answer. The first question is: 'Is HashMap thread-safe?' The expected answer is: 'No, HashMap is not thread-safe.' Step 2 — The question goes to the AI. The EvalRunner sends the question to EngineerPrep's AI tutor — the same Spring @Service that real users call. Nothing special about the call; it's the exact same code path. Step 3 — The answer comes back. The model replies: 'HashMap is not thread-safe. Use ConcurrentHashMap for concurrent access.' Step 4 — The checker compares. A component called the judge receives two things: the model's answer and the expected answer.…
We'll build a minimal eval runner as a Spring Boot @Service . It has one job: take a question, call the AI tutor, and check whether the answer contains the expected key phrase. Here is the plan line by line. The EvalCase record holds one test case: the question and the phrase we must see in the answer. A Java record is just a simple data container — think of it as a struct. The TutorEvalService class is a Spring service. The @Service annotation tells Spring to manage it. The constructor receives a ChatClient — that is Spring AI's interface for sending messages to the language model, backed by Amazon Bedrock and Claude inside EngineerPrep. This is constructor injection : Spring hands the dependency in through the constructor rather than setting a field directly. It's the recommended style because it makes dependencies explicit and easier to test.…