By the end of this lesson you will understand what tokens are, why every language model thinks in tokens instead of characters or words, and how that knowledge directly shapes the code you write when you build AI features in Java.
Imagine you are on-call for EngineerPrep. A user files a bug: "The AI tutor cut off my answer halfway through." You pull up the logs. The HTTP request completed successfully. Spring returned a 200. Bedrock (the AWS service that runs the Claude language model) returned a response. Nothing threw an exception. So why is the answer half-finished? You dig a little deeper and spot a number in the Bedrock response metadata: stop reason: max tokens . The model hit a limit — but a limit measured in tokens , not characters, not words, not sentences. What even is a token? And why does every language model care about them so much? That is exactly what this lesson answers.
Think about how you learned to read as a child. Your teacher did not make you memorize every possible word all at once. Instead, you learned small, reusable pieces — letters, then common letter groups like "ing" or "tion" — and you combined them to read any word you had never seen before. A language model does something very similar. Instead of working with individual characters (too small, too many steps) or full words (too many unique possibilities), it works with tokens — small chunks of text that sit somewhere between the two. A token is the smallest unit of text that a language model processes. Think of it as one puzzle piece. A sentence is built from many puzzle pieces snapped together. Here are a few examples to make it concrete: - The word "running" might be split into two tokens: "run" and "ning" . - The word "Java" might be exactly one token.…