LLM Foundations

LLM foundations for engineers: what tokens and context windows really cost, how sampling shapes an answer, and why a prompt succeeds twice, then fails.

Lessons in this topic

• What Is a Large Language Model?
• LLM Strengths & Limitations
• Tokens: The Basic Unit of Text
• Tokens: The Basic Unit of LLM Text
• How LLMs Read Text (Tokenization)
• Tokenization & Encoding Schemes
• Embeddings vs Tokens
• Context Windows: Capacity & Limits
• Context Windows
• Next-Token Prediction
• Next-Token Prediction & Probability Distributions
• Probability Distributions
• Temperature
• Top-k & Top-p Sampling
• Why LLMs Hallucinate