Context Is All You Have: How LLM Attention Actually Works

You've seen the marketing: "128k context window!" "1 million tokens!" But what does that actually mean for your use case? And why does your chatbot still forget what you said 20 messages ago? This is the first post in a series on LLM internals — no hype, no doomerism, just the mechanics that determine whether your AI application works or falls apart. The Attention Mechanism (30 Second Version) Every modern LLM is built on transformers. The core operation is attention: for each token the model generates, it looks back at every previous token and decides how much to "attend" to each one. Mathematically: Attention(Q, K, V) = softmax(QK^T / √d) × V In plain English: the model converts your input into queries (Q), keys (K), and values (V). It computes similarity scores between queries and keys, normalizes them with softmax, and uses those scores to weight the values. The key insight: attention is O(n²) in sequence length. Double your context, quadruple the compute. This is why context windo