Module 2 of 5 · Novice · 8 min read

How language models work

Tokens in, tokens out. A plain-language tour of how a model turns your question into an answer.

Key takeaways

1Models read and write in tokens, chunks of roughly three-quarters of a word.
2At its core, a language model predicts the next token, over and over.
3Training has two big phases: pre-training on vast text, then post-training to make the model helpful and safe.

01Tokens and prediction

A language model does not see letters or words the way you do. Text is split into tokens: common words are one token, rare words are several. The model's job is to predict which token should come next, given everything before it.

It generates an answer by predicting one token, adding it to the text, and predicting again, hundreds or thousands of times. That simple loop, run on a model with hundreds of billions of parameters, produces fluent writing, reasoning and code.

Example

"The capital of France is" → the model assigns a very high probability to " Paris" and a tiny one to everything else.

02Pre-training and post-training

In pre-training, the model reads a large slice of the public internet, books, code and other data, learning to predict the next token. This is where it picks up grammar, facts, reasoning patterns and much more. It costs tens or hundreds of millions of dollars in computing.

Post-training turns that raw predictor into an assistant. Techniques like reinforcement learning from human feedback (RLHF) and reinforcement learning on verifiable tasks teach it to follow instructions, reason step by step, use tools and decline harmful requests.

03The context window

The context window is everything the model can see at once: your messages, its replies, any documents and tool results. Frontier models in 2026 handle hundreds of thousands to millions of tokens, enough for whole codebases or several books.

The model has no memory beyond this window unless a product adds one. If something is not in the context, the model does not know it, which is why giving good context is the heart of prompting.

Knowledge cutoff: the model only knows what was in its training data up to a date.
Tools like web search fill the gap for recent information.