You will understand what next-token prediction is, why it forces every LLM interaction to be measured in tokens, how probability distributions govern every word the model outputs, and how that shapes every architectural decision EngineerPrep makes around its AI tutor, lesson pipeline, and semantic…
It is 11 PM. You are on-call. A user files a support ticket: they pasted their entire system-design interview study guide into the EngineerPrep AI tutor chat and received a cold error. The guide is 8,000 words. The Claude model you are running on Amazon Bedrock advertises a 100,000-token context window. Eight thousand words should be nowhere near that limit. You pull the request log. The tokenizer counted 14,200 tokens. The document was 8,000 words but 14,200 tokens. You trimmed content that felt redundant, got it down to 7,000 words — still 12,400 tokens. You are not dealing with a counting bug. You are dealing with a fundamental mismatch between the unit humans think in (words) and the unit the model reasons in (tokens). EngineerPrep hit this exact wall when building the lesson-generation pipeline.…
The on-call story above is really a story about the atomic unit of LLM reasoning. One sentence recap: the model could not see the whole document because its entire architecture is built around processing and predicting one token at a time. What exactly is this? Next-token prediction is the task an LLM is trained to perform: given a sequence of tokens so far, output a probability distribution over every token in the vocabulary, then sample or select the next token from that distribution. Do it again. Repeat until the model emits an end-of-sequence token. That is the entire generation loop. There is no sentence-level planner, no paragraph-level reasoner. There is only: what is the single most likely (or interestingly probable) next token given everything that came before? A token is not a word. It is the smallest unit the model's vocabulary knows about…
Scene Setup Imagine a horizontal conveyor belt stretching left to right across the screen. On the belt sit colored tiles, each one a token — a short string like 'the', 'Java', 'inter', ' face', '.', '\n'. The belt represents the context window: everything the model has seen so far. To the right of the belt sits a large vertical bar chart — the model's vocabulary. Each bar represents one of, say, 50,000 possible next tokens. Right now all bars are the same height: the model knows nothing yet. Animation step 1 — feed in the prompt Tiles start loading onto the belt from the left: 'Explain', 'how', 'a', 'Spring', 'Boot', '@Service', 'bean', 'is', 'created'. Each tile slides onto the belt and lights up briefly as it lands. After the last tile lands, pause. The belt is now holding nine tokens. The vocabulary bar chart on the right is still flat — we have not asked the model to predict yet.…
From raw text to token IDs Before the model sees anything, the tokenizer — a separate, deterministic program — converts a string into a list of integer IDs. Anthropic's Claude uses a variant of BPE. The BPE algorithm was trained on a large corpus: it starts with individual bytes, then iteratively merges the most frequent adjacent pair into a new token, repeating until the vocabulary reaches its target size (typically 32,000–100,000 entries). The result is a trie-like merge table. Tokenization is O(n log n) in the number of merge operations but fast in practice — a 10,000-token document tokenizes in under 5 ms on the JVM. The token IDs are the only thing the model ever sees. 'Java' might be ID 14507. '@Service' might be IDs 31, 5236, 12 — three tokens. Case matters, whitespace matters, punctuation matters. 'java' and 'Java' are different IDs with different embedding vectors.…