Automatic metrics are simple, computable scores that measure how close a generated text is to a known-good answer — no human reading required — and they are a practical foundation for any AI system you want to improve systematically over time.
Picture this: the EngineerPrep team ships a new version of the AI tutor powered by a different Claude model on Amazon Bedrock. The new model is cheaper and faster. But is it better? You open a handful of answers and they look okay. Your teammate checks a few others and shrugs — 'seems about the same.' That's the problem. 'Seems about the same' is not something you can put in a pull request or a Jira ticket. It doesn't tell you whether to ship. It doesn't catch a regression at 2 AM. Your first instinct might be: hire humans to rate every response. That works — once. It costs time and money, and you can't do it every time you change a prompt or swap a model. So the question is: can a computer score the quality of text automatically, without a human in the loop? That's exactly what automatic metrics are for — and by the end of this lesson you'll know how to use them.
An automatic metric is a formula that takes two pieces of text and returns a number. One piece of text is the generated output — what the AI actually said. The other is the reference — what a correct answer looks like. The metric compares them and gives you a score, usually between 0 and 1. Higher means closer to the reference. Think of it like a spell-checker. A spell-checker doesn't understand your writing — it just compares your words against a dictionary. Automatic metrics work the same way: they don't understand meaning, they measure similarity. The most widely used automatic metric is called BLEU (Bilingual Evaluation Understudy — the name comes from machine translation research, but it's used across many NLP tasks). BLEU measures how many short word sequences from the generated text also appear in the reference, weighted by a brevity penalty to discourage very short outputs.…
Imagine the EngineerPrep AI tutor generates this answer to a learner's question about Spring beans: Generated: 'A Spring bean is an object managed by the Spring container.' And the reference — the known-good answer written by an engineer — is: Reference: 'A bean is an object that the Spring container creates and manages.' Step 1 — Split into words. Both sentences become lists of individual words (this is called tokenisation — just chopping text into pieces). - Generated words: A, Spring, bean, is, an, object, managed, by, the, Spring, container - Reference words: A, bean, is, an, object, that, the, Spring, container, creates, and, manages Step 2 — Find the overlapping words. We look for unique words that appear in both lists: A , bean , is , an , object , the , Spring , container — eight unique word types match.…