How an LLM Generates an Answer: From Text to the Next Token
An LLM turns text into integers (tokens), maps tokens to high-dimensional vectors, processes them through stacked transformer attention layers to extract context, and computes a probability distribution over its vocabulary to predict one single token at a time.
Why It Matters
Understanding this step-by-step pipeline demystifies artificial intelligence. It explains why models hallucinate, why context windows cost money, and why sampling parameters like temperature shape model behavior.
Simple Plain-English Analogy
Imagine an ultra-fast autocomplete system that has read millions of books. It does not understand thoughts like a human. Instead, it looks at the sequence of words written so far and calculates the most statistically coherent next word, appends it, and repeats the process thousands of times.
Visual Architecture Flow
Prompt Text ("The sky is")
↓
[Tokenizer]: Breaks into integers [464, 6766, 318]
↓
[Embedding Matrix]: Converts IDs into 4096-dimensional vectors
↓
[Positional Encoding]: Injects coordinate position using RoPE
↓
[Transformer Blocks (x32)]: Self-Attention routes context between words
↓
[RMSNorm & Final Linear Head]: Projects 4096-dim vector to 128,000 logits
↓
[Softmax & Temperature]: Converts logits into probabilities summing to 1.0
↓
[Sampling (Top-K / Top-P)]: Picks "blue" (84%)
↓
Append "blue" to context and repeat forward pass for next word!Step-by-Step Technical Breakdown
Tokenization
Your raw text string is broken into subword tokens and mapped to numerical IDs using Byte-Pair Encoding (BPE).
Embedding Lookup & Positional Encoding
Each token ID looks up a dense floating-point vector in the embedding matrix. Rotary Position Embeddings (RoPE) rotate vectors to encode order.
Multi-Head Self-Attention
Queries, Keys, and Values compute similarity dot products, enabling words to update their meaning based on surrounding context.
Feed-Forward Transformation
Non-linear MLP layers (SwiGLU) retrieve associative factual knowledge learned during pretraining.
Logits & Softmax Normalization
The output vector is projected across the entire vocabulary to yield raw logits, which Softmax normalizes into percentages.
Sampling & Autoregressive Repeat
Temperature and Top-P filter candidates. One token is chosen, appended to the prompt, and fed back into the model.
Practical Real-World Example
Suppose you type: "The capital of France is" 1. Tokens: ["The", " capital", " of", " France", " is"] 2. After 32 layers of attention, the final vector attends heavily to "capital" and "France". 3. The vocabulary projection assigns: - "Paris": logit 18.2 (Prob: 98.2%) - "Lyon": logit 11.4 (Prob: 0.8%) - "a": logit 9.1 (Prob: 0.1%) 4. The model samples "Paris", appends it, and repeats to predict the closing period.
Experience This In The Burnjet Laboratory
Test parameters and inspect intermediate calculations live.
- •Believing the model writes the whole paragraph at once rather than token by token.
- •Assuming the model has real-time internet search unless explicitly connected to a retrieval tool.
- •Thinking temperature alters model weights rather than just shaping the sampling distribution.
What happens immediately after an LLM selects a new token during text generation?
Key Takeaways
- LLMs are autoregressive next-token predictors.
- All text is converted to token IDs and continuous vector embeddings.
- Self-attention layers dynamically blend context between tokens.
- Softmax produces probabilities that guide stochastic or greedy sampling.
Frequently Asked Questions
Why does an LLM give different answers to the exact same prompt?
When temperature is greater than 0, the model samples probabilistically from the candidate distribution rather than always picking the top logit.
Does the model learn new facts while I am chatting with it?
No. Inference uses frozen, read-only weights. Any temporary "memory" lives strictly inside your current context window.