Top-K: The Fixed Candidate Pool
|
What Is Top-K?
Top-K sampling limits the model’s next-token selection to only the K most likely tokens from the full vocabulary. All tokens outside the top K are completely excluded — they have zero probability of being selected.
| MENTAL MODEL | Top-K is like a talent show with a fixed number of contestants. If K = 50, only the 50 most likely next words are allowed on stage. The model then picks from those 50 (influenced by Temperature). Everyone else is sent home regardless of how plausible they might be. |
Top-K: How It Works (Step by Step)
- Model generates probability scores for all ~50,000 tokens in vocabulary
- Probabilities are sorted in descending order
- Only the top K tokens are kept; all others are set to zero
- Temperature is applied to the remaining K tokens to shape the final selection
- One token is selected from the filtered, temperature-scaled distribution
Top-K Example
| Rank | Word | Probability | Top-K=3 Status |
| 1 | cat | 0.25 | ✓ INCLUDED |
| 2 | dog | 0.20 | ✓ INCLUDED |
| 3 | hat | 0.15 | ✓ INCLUDED |
| 4 | car | 0.10 | ✗ EXCLUDED |
| 5 | banana | 0.05 | ✗ EXCLUDED |
| 6+ | all others | <0.05 | ✗ EXCLUDED |
Top-K vs Temperature: What’s the Difference?
| Aspect | Temperature | Top-K |
| What it controls | How the probabilities are weighted/scaled | How many candidates are available for selection |
| Effect of low value | Strong weighting to top token — predictable | Fewer candidates — more controlled |
| Use case | Adjust creativity level | Set a fixed ceiling on token variety |
| Typical values | 0.0–1.0 (or 2.0) | 10–100 (varies by model) |