How LLMs Work — Interactive Learning Lab
Learn how language models process text, tokenize input, create continuous vector representations, use self-attention, predict probability distributions, and generate coherent text one token at a time.
Step-by-Step Computational Dataflow
Text
Raw human strings and code sequences entered by the user.
Tokens
Subword segments created by Byte-Pair Encoding (BPE).
Token IDs
Unique integers mapping each subword to the model vocabulary dictionary.
Embeddings
Dense high-dimensional semantic vectors (e.g. 4096 dimensions).
Position Info
Rotary Position Embeddings (RoPE) rotating vectors to encode sequence order.
Attention
Queries, Keys, and Values routing contextual focus across tokens.
Transformer
Stacked blocks with RMSNorm, Multi-Head Attention, and SwiGLU FFNs.
Logits
Raw unnormalized real-numbered scores across all 128,000 vocabulary tokens.
Probabilities
Softmax exponentiation and normalization scaling candidates to 100%.
Next Token
Sampled candidate token chosen via Temperature, Top-K, and Top-P.
Repeat
Newly sampled token is appended to context, starting the next forward pass.
Generated Text
Complete streaming answer returned to user upon reaching <|end_of_text|>.
How an LLM Generates an Answer: From Text to the Next Token
Read our original 15-part illustrated guide exploring the exact mathematics of transformer decoders, floating-point matrix multiplications, and probability sampling.