Module 1 of 6 · Expert · 10 min read

Transformers and mixture-of-experts

Attention, embeddings, scaling laws and why most frontier models are now sparse.

Key takeaways

1Self-attention lets every token weigh every other token in context.
2Mixture-of-experts activates only a fraction of parameters per token.
3Scaling compute, data and test-time reasoning all improve capability.

01Attention

Each token is turned into a vector (an embedding). In every layer, self-attention computes how much each token should draw on every other token, using learned query, key and value projections. Stacking dozens of layers builds rich, contextual representations.

Attention cost grows with the square of sequence length, which is why long context needs engineering tricks such as sparse or linear attention, sliding windows and efficient KV caches.

02Mixture-of-experts

In an MoE layer, a router sends each token to a few of many expert sub-networks. A model can have over a trillion total parameters while activating only tens of billions per token, giving large-model knowledge at smaller-model cost.

DeepSeek, Kimi, Qwen, Mistral and most frontier labs use MoE designs. The trade-off is memory: all experts must be loaded even though few are used per token.

03Three ways to scale

Progress has come from three compounding axes.

Pre-training compute and data: bigger models on more tokens.
Post-training: reinforcement learning on verifiable tasks such as math, code and agentic environments.
Test-time compute: letting models think longer before answering.

Insight

Since 2024, RL post-training and test-time compute have driven many of the largest jumps on reasoning benchmarks.