You will understand exactly how tokenization works from raw bytes to BPE merge tables, why token counts diverge from word counts in ways that matter for cost and latency, and how to control it in a production Spring Boot system built on Amazon Bedrock.
Every LLM has a fixed vocabulary — not of words, but of tokens . A token is the smallest unit of text the model operates on. It might be a whole word ( hello ), a sub-word fragment ( ing , ization ), a single character, or even a raw byte. The model never sees your string; it sees a sequence of integer IDs, one per token, looked up from a vocabulary table. Why not just split on spaces? Space-splitting breaks immediately on contractions ( don't → don , 't ), on morphology ( running , runner , ran all share a root), on code ( System.out.println( is one logical unit), and on non-Latin scripts that have no spaces at all. Vocabulary size also matters: if every word is a token you need hundreds of thousands of IDs; if every character is a token your sequences get too long for the context window. Byte-Pair Encoding (BPE)…
The scene: a whiteboard with a tiny corpus Imagine the screen shows four short training sentences written in a column on the left side of a whiteboard: low low low low lower lower new new new newer newer Step 1 — Start with characters + end-of-word marker An animation splits every word into individual characters. Each word also gets a special </w marker appended to signal word boundaries (important for later reconstruction). The whiteboard now shows a frequency table: l o w </w → 4 l o w e r </w → 2 n e w </w → 3 n e w e r </w → 2 Step 2 — Count every adjacent pair A glowing highlight sweeps through each sequence counting bigrams. The most frequent pair lights up in yellow: e r appears 4 times (twice in lower , twice in newer ). An arrow draws itself from the bigram table to a new "Merges" list on the right side of the whiteboard: Merge 1: e r → er . Step 3…
The three data structures every BPE tokenizer carries 1. Vocabulary map — Map<String, Integer from token string to integer ID. Size is fixed after training (~50k for GPT-2, ~100k for Claude-class models). 2. Merges list — an ordered List<Pair<String,String of merge rules, highest priority first. Ordering matters: the tokenizer applies them in training order. 3. Inverse vocabulary — Map<Integer, String for decoding token IDs back to strings. Encoding pass — step by step Given input string "pgvector" , a tiktoken-style encoder does this: Step 1 — Pre-tokenization (regex split) Before BPE runs, a regex splits the raw string into 'words' (roughly: sequences of letters, sequences of digits, punctuation). This matters because don't splits into don, ', t before BPE sees it, so no merge can ever bridge that boundary. Claude and GPT-4 use similar but not identical regexes…