Top-P (Nucleus Sampling): The Dynamic Candidate Pool
Learning Objectives:
|
What Is Top-P?
Top-P, also called Nucleus Sampling or Softmax Sampling, selects from the smallest set of tokens whose combined probability adds up to P (expressed as a value between 0 and 1). Unlike Top-K (which uses a fixed number of candidates), Top-P uses a dynamic number of candidates that adapts to the model’s confidence.
MENTAL MODEL | Top-P is like a guest list where you keep inviting guests until the party is ‘full enough.’ If the model is very confident (one word has 90% probability), the nucleus is just 1 word. If the model is uncertain, the nucleus expands to include more options until together they cover P% of the probability mass. |
Top-P: How It Works (Step by Step)
- Model generates probability scores for all tokens, sorted descending
- Calculate cumulative sum: keep adding probabilities from top to bottom
- Stop when the cumulative sum reaches P (e.g., P = 0.9 = 90%)
- Only tokens in this ‘nucleus’ are eligible for selection
- Temperature is then applied within the nucleus before final selection
Top-P Example (P = 0.90)
Rank | Word | Probability | Cumulative | In Nucleus (P=0.9)? |
1 | cat | 0.25 | 0.25 | ✓ Yes |
2 | dog | 0.20 | 0.45 | ✓ Yes |
3 | hat | 0.15 | 0.60 | ✓ Yes |
4 | car | 0.10 | 0.70 | ✓ Yes |
5 | banana | 0.05 | 0.75 | ✓ Yes |
6 | tree | 0.05 | 0.80 | ✓ Yes |
7 | robot | 0.04 | 0.84 | ✓ Yes |
8 | moon | 0.03 | 0.87 | ✓ Yes |
9 | star | 0.02 | 0.89 | ✓ Yes |
10 | jazz | 0.01 | 0.90 | ✓ Yes — STOP |
11+ | others | <0.01 | >0.90 | ✗ Excluded |
Top-K vs Top-P: When to Use Which
Scenario | Recommended | Why |
You want a consistent, fixed vocabulary range | Top-K | Predictable number of candidates every time |
You want output to adapt based on model confidence | Top-P | Dynamic — expands/contracts with certainty level |
Creative writing with varied topics | Top-P | Adapts to uncertainty — better across diverse content |
Code generation (narrow vocabulary) | Top-K or low Top-P | Tight control on which tokens are possible |
Most production LLM APIs (default) | Top-P = 0.95 default | Industry standard balanced default |