How LLMs Generate Text: The Foundation

How LLMs Generate Text: The Foundation

Learning Objectives:

(1) Explain how LLMs predict the next token.

(2) Define a token and understand the tokenization process.

(3) Explain why understanding token generation unlocks parameter control.

How LLMs Actually Work (Simplified)

Language models do not ‘think’ or ‘understand’ in the human sense. They are statistical text prediction engines. Given a sequence of tokens (words/sub-words), the model assigns a probability to every possible next token in its vocabulary, then selects one.

This selection process is where ALL control parameters operate. When you change Temperature, Top-K, or Top-P, you are changing HOW the model selects from those probability distributions.

What Is a Token?

A token is the fundamental unit of text that an LLM processes.

Tokens are not always whole words — they can be sub-words, punctuation, or even single characters.

try https://platform.openai.com/tokenizer to test tokens. 

Why Tokens Matter for Prompt Engineers

• Cost: Most LLM APIs charge per token. Longer prompts + longer outputs = higher cost.
• Context window: Every LLM has a maximum context window (e.g., 128K tokens). Exceeding it truncates your input or output.
• Max Response: The parameter that caps your output is measured in tokens, not words.
• Efficiency: A concise, well-structured prompt uses fewer tokens while producing better output than a verbose, vague one.

The Probability Distribution: Where Parameters Live

After processing your prompt, the model generates a probability score for every word in its vocabulary (50,000+ words). Before selecting the next word, it applies your control parameters to filter and shape that distribution. Understanding this is the key to understanding all parameters.

Typical Cost Example

Suppose your application uses:

• 1 million input tokens
• 500,000 output tokens