Top-k and top-p are two simple controls that shape how an LLM picks its next word — understanding them lets you tune any AI feature from robotic and repetitive to natural and reliable.
You ship an AI tutor feature. Users love it — for a day. Then the bug reports come in. 'It keeps saying the same thing.' Or worse: 'It said something completely made up and wrong.' Both complaints trace back to the same root cause: the model is choosing its next word badly. Your first instinct is probably 'I'll just change the prompt.' That's the natural guess. But the prompt isn't the problem. The problem is a setting you haven't touched yet — one that controls how the model samples (picks) each word from a list of candidates. Get that setting wrong and your AI is either a parrot or a chaos engine. Get it right and it sounds like a thoughtful human. So — what exactly is happening when a model picks its next word, and which knob do you turn to fix it?
Every time an LLM (Large Language Model — an AI trained on text) is about to write the next word, it doesn't just pick one word. It produces a list of every word it knows, each with a probability — a score saying 'how likely is this word to come next?' Think of it like a bakery display case. There are hundreds of items behind the glass. The chocolate croissant has a big label saying '40% chance you want this.' The plain bagel says '25%.' The mystery loaf at the back says '0.001%.' You could always pick the most popular item. But if you do that every single time, every customer gets the same order. Boring — and that's exactly the robotic repetition bug from the hook. Alternatively, you could pick completely at random from all hundreds of items. Then someone occasionally walks out with the mystery loaf. That's the nonsense bug.…
Imagine the model just wrote the words 'A deadlock occurs when two threads' and now it needs the next word. Step 1 — The full probability list The model produces scores for every word it knows. Picture a tall bar chart with thousands of tiny bars. A few bars are tall; most are nearly invisible. The tallest bars (most likely next words) might be: - 'each' → 38% - 'both' → 22% - 'simultaneously' → 14% - 'forever' → 9% - 'cannot' → 7% - 'try' → 5% - ... (thousands more, all below 1%) If you always picked the tallest bar ('each'), every answer would be identical. If you sampled from all thousands of bars, you'd occasionally pick something absurd. Step 2 — Apply top-k (k=4) Slice off everything except the top 4 bars. The chart now shows only: - 'each' → 38% - 'both' → 22% - 'simultaneously' → 14% - 'forever' → 9% All thousands of tiny bars are gone.…