Frequency Penalty & Presence Penalty: Fighting Repetition
Learning Objectives: (1) Define Frequency Penalty and Presence Penalty. (2) Distinguish between the two penalties and know when to use each. (3) Predict output behavior at high penalty values. |
Why Do LLMs Repeat Themselves?
LLMs are trained to predict statistically likely next tokens. If a word appeared frequently in the context window (your conversation so far), the model assigns higher probability to it appearing again. Without intervention, this creates repetitive, circular outputs — especially in long-form generation.
Frequency Penalty
- What it does: Reduces the likelihood of a word being repeated based on how frequently it’s already appeared in the text.
- Example: If “sun” has been mentioned 3 times already, a frequency penalty will make the model less likely to say “sun” again.
- Use case: Helps avoid repetitive wording, especially in longer texts.
- Think of it as: “The more you say it, the less likely you are to say it again.”
Effect | Use When | |
0 (default) | No penalty — tokens can repeat freely | Short outputs where repetition isn’t a concern |
0.5–1.0 | Moderate discouragement of repeated words | Blog posts, summaries, general content |
1.5–2.0 | Strong reduction of any repeated token | Long-form content, creative writing, listicles |
Presence Penalty
- What it does: Penalizes words that have already appeared at all, regardless of how often.
- Example: If “sun” has appeared even once, presence penalty reduces the chance it shows up again.
- Use case: Encourages more diverse word choices by discouraging reuse of any previously used words.
- Think of it as: “If you’ve said it once, try something new.”
Value | Effect | Use When |
0 (default) | No penalty on previously used tokens | Consistent, focused content on one topic |
0.5–1.0 | Gently encourages introducing new ideas | Multi-topic brainstorms, diverse content sets |
1.5–2.0 | Strong push to introduce new concepts/vocabulary | Ideation, exploratory writing, wide-ranging reports |
Frequency vs Presence Penalty: Key Distinction
FREQUENCY PENALTY | Targets HOW OFTEN a token repeats. Good for: reducing word-level repetition (‘the’, ‘very’, specific nouns used too much). |
PRESENCE PENALTY | Targets WHETHER a token has appeared at all. Good for: encouraging topic diversity, making the model explore new ground in a long piece. |
WARNING | Setting both penalties too high (>1.5) simultaneously can degrade output quality — the model starts avoiding even necessary words, producing grammatically odd or factually incomplete text. Use with caution. |