Module 10 of 11 · Expert · 10 min read

Long context, context engineering and memory

Decide what goes into the context window and where, and how agents remember across steps and sessions.

Key takeaways

1The context window is working memory: everything in it competes for attention.
2Models can miss information in the middle of long inputs (Liu et al., 2023), so placement matters.
3Context engineering means curating, ordering, compressing and refreshing what the model sees.
4Long-term memory is stored text you choose to bring back. Design what gets saved.

01Bigger windows, same discipline

Context windows have grown enormously, so you can often paste entire codebases, contracts or books. But more context is not automatically better: irrelevant material dilutes attention, costs money and slows responses. Liu et al. (2023), in 'Lost in the Middle', found that models used information at the start and end of long inputs more reliably than information in the middle.

02Context engineering

The Agent Builder path has a full module on context engineering for agents; these are the prompting essentials.

Select: include what the task needs and leave out what it doesn't.
Order: long documents first, instructions and question last; the most important material near the start or end.
Label: wrap each item in tags with a title and source, so it can be referenced.
Compress: summarise old conversation turns and tool outputs instead of keeping them verbatim.
Refresh: re-insert the goal and key constraints in long sessions.

03Compaction for long sessions

In long conversations and agent runs, periodically replace the history with a structured summary. Do it deliberately, with a prompt that preserves what matters, rather than letting an app silently drop old messages.

Example

Summarise our work so far so we can continue in a fresh session. Include: - The goal and constraints, exactly as agreed. - Decisions made and why. - Current state: what is done and what is in progress. - Open questions and next steps. - Any facts, names or numbers we will need again. Leave out dead ends unless they explain a decision.

04Kinds of memory

Every kind of memory is ultimately text stored somewhere and selectively put back into the prompt. The design questions are what to save, when to retrieve it, and when to forget it.

Working memory: the current context window.
Session memory: summaries and notes carried through a long task.
Long-term memory: facts and preferences saved across sessions and retrieved when relevant.
Episodic memory: records of past attempts and lessons, as in Reflexion.

05The progress file pattern

For long agent tasks, ask the agent to keep a progress file (for example NOTES.md) with the goal, plan, decisions and status, and to read it at the start of each step. It survives context resets, is easy for you to inspect, and keeps the run anchored to the original plan.

Tip

Tell the agent exactly what the notes file must contain and to keep it short. A notes file that grows without limit becomes the context problem it was meant to solve.

06Memory and privacy

Memory features remember what you tell them. Review what an assistant has saved about you, delete what you don't want kept, and avoid putting sensitive personal or business details into tools with memory unless you understand where they are stored.