Agentic AI · Module 12 of 41 · 10 concepts · about 16 min

Context Engineering

Deciding what the model reads on each turn: the pieces that compete for space, how to budget tokens between them, and how to select, compress and summarise so that a long task stays affordable and the model stays on course.

At work: The work that replaced 'prompt engineering' as agents got longer: most quality, cost and speed problems in a production agent are decided by what is put in front of the model, and in what order.

What context engineering is

What it is

The context is everything the model reads on one call. Context engineering is deciding, for each turn, what goes in it. It differs from prompt engineering in scope:

Prompt engineering is writing the instructions well.
Context engineering is assembling everything: instructions, tools, memories, documents, history and intermediate results, chosen and arranged afresh on every turn.

Why it matters

A model's decision can only be as good as what it was shown.

Missing information produces a guess.
Too much information buries the part that matters, and is paid for on every turn.
Contradictory information produces unpredictable behaviour.
In a twenty-step task the context is rebuilt twenty times, so small choices compound.

In the real world: A coding agent's quality problem turned out not to be the model: forty old file listings were crowding the context. Removing stale ones fixed it without touching a prompt.

Thinking of the context as an append-only log. Nothing forces you to resend everything that ever happened. The context is assembled by your code each turn, and you may select, trim, reorder and summarise.

The eight things that compete for the context

What it is

Every agent context is built from the same eight kinds of content:

System prompt: the standing role and rules.
Instructions for this task: the goal and any constraints.
Tool descriptions: every offered tool's name, purpose and schema.
Memory: the user's profile and relevant past facts.
Retrieved documents: passages fetched for this question.
Conversation history: earlier messages in this session.
Examples: sample inputs and outputs showing the wanted behaviour.
Intermediate results: tool outputs, the plan, notes made so far.

Why it matters

Naming them turns a vague 'the prompt is too long' into a decision about which part to shrink.

Tool descriptions and history usually take the most space.
Intermediate results grow fastest.
The system prompt and tool definitions are the same every turn, which makes them ideal for caching.

In the real world: A team prints a per-category token count for a typical run and finds tool results are 87 percent of the input; trimming them does more than a month of prompt edits.

Visual example

Contents

The last two columns tell you how to treat each.

What fills an agent's context

content	changes	grows with task length	treatment
System prompt	Never	No	Cache
Tool descriptions	Rarely	No	Cache; offer fewer
Task instructions	Per task	No	Keep; restate late
Memory	Per turn	No	Retrieve top few
Retrieved documents	Per turn	Somewhat	Grade and trim
Conversation history	Per turn	Yes	Summarise old turns
Examples	Rarely	No	A few, well chosen
Intermediate results	Per turn	Yes, fastest	Trim; store outside; fetch on demand

Optimising the system prompt's wording while tool results take up nine-tenths of the context. Measure where the tokens are before editing anything.

Token budgeting

What it is

Token budgeting is giving each kind of content a share of the context, in advance:

choose a working budget well below the model's maximum;
reserve room for the model's reply and its reasoning;
give each category a ceiling;
when a category exceeds its ceiling, apply its rule: drop the oldest, summarise, or retrieve fewer.

Why it matters

Without a budget the context fills with whatever arrived first.

A budget makes cost per turn predictable.
It forces a decision about priorities before a crisis: if space is short, memories shrink before the rules do.
Staying well under the maximum keeps quality up: models attend less reliably to very long inputs.

In the real world: A support agent runs on a working budget under ten thousand tokens although its model accepts a hundred times that; replies are faster, cheaper and more accurate than when the window was filled.

Visual example

Budget

Working budget 8,500 input tokens, with 4,000 more reserved for the reply.

A token budget for one turn of a support agent

content	ceiling (tokens)	share	rule
System prompt and rules	800	9%	Fixed; cached
Tool definitions	1,500	18%	Fixed; cached; only the tools this task needs
User profile and memories	400	5%	Top three memories, not all of them
Retrieved documents	2,000	24%	After grading; trimmed to relevant passages
Summary of earlier turns	600	7%	Replaces the old history
Recent turns, verbatim	3,000	35%	The last few steps in full
Goal and plan reminder	200	2%	Restated at the end, every turn

Total 8,500 tokens. The model's window is far larger; the budget is set by need, not by the limit.

BUDGET = {"system": 800, "tools": 1500, "memory": 400, "documents": 2000,
          "summary": 600, "recent": 3000, "reminder": 200}

def fit(parts: dict[str, list[str]]) -> dict[str, list[str]]:
    """Trim each category to its ceiling. Newest items are kept first."""
    fitted = {}
    for name, items in parts.items():
        kept, used = [], 0
        for item in reversed(items):                 # newest first
            size = count_tokens(item)
            if used + size > BUDGET[name]:
                break
            kept.insert(0, item)
            used += size
        fitted[name] = kept
    return fitted

Budgeting to the model's maximum window. A context that technically fits can still be too long to use well, and leaves no room for the reply. Budget to what the task needs.

Context window management

What it is

Context window management is keeping a long task inside its budget as it runs. The context of a loop grows every step, because each step adds a tool request and its result, and everything is resent. Four tools control it:

Trim tool results before they enter the history.
Clear old tool results that are no longer needed.
Summarise (compact) old turns.
Offload: store bulky results outside the context and keep only a reference.

Why it matters

Growth is the central cost fact of agents.

Every token added at step 2 is paid for again at steps 3, 4, 5 and every later step.
So total cost rises much faster than the number of steps.
Trimming what each step adds is the single most effective cost control.

In the real world: A research agent that returned whole web pages as tool results hit its window at step nine; returning extracted paragraphs instead lets it run forty steps for less money.

Visual example

Growth

Fixed start of 2,500 tokens. Raw results add 2,500 per step; trimmed results add 400.

Input tokens over a 12-step task, raw versus trimmed tool results

	raw results	trimmed results
Input at step 1	2,500	2,500
Input at step 12	30,000	6,900
Total input over the task	195,000	56,400
Cost of the task (medium model)	₹51	₹16.35

Trimming cuts the task's cost by 68 percent with no change to the model or the prompt.

Waiting until the context overflows and then truncating from the top. That deletes the system prompt and the original goal first, which are the two things the agent most needs.

Practise it: Design a Self-Correcting Code Agent

Dynamic context selection

What it is

Dynamic context selection means choosing, each turn, only what this turn needs:

Tools: offer the subset relevant to the current stage.
Memories: retrieve the top few by relevance, not the whole profile history.
Documents: fetch on demand through a search tool, instead of pre-loading.
Examples: pick the few most similar to the current input.
Instructions: load detailed guidance for a sub-task only when that sub-task starts. Often called skills or progressive disclosure.

Why it matters

Relevance beats completeness.

A model shown five relevant items uses them. Shown fifty, it uses some at random.
Loading on demand keeps the fixed part of the context small, so every turn is cheaper.
The agent can still reach everything; it simply has to ask.

In the real world: An agent with a hundred pages of procedures loads only a one-line index of them, and reads a full procedure when it decides it needs one; its fixed context shrinks by nine-tenths.

def build_context(state, user) -> dict:
    stage_tools = TOOLS_BY_STATE[state.stage]                     # only this stage's tools
    memories = recall(user.id, query=state.goal, k=3)             # top three, not all
    return {
        "system": SYSTEM,                                         # stable: cacheable
        "tools": [TOOL_DEFS[name] for name in stage_tools],
        "messages": [
            {"role": "user", "content": render_profile(user) + render(memories)},
            *summarised(state.history),                           # old turns compressed
            *state.history[-6:],                                  # recent turns verbatim
            {"role": "user", "content": reminder(state)},         # goal and plan, last
        ],
    }

Changing the tool list or system prompt on every turn. That defeats prompt caching, which depends on the start of the context being identical between calls. Vary the end of the context, keep the start stable.

Context compression and summarisation

What it is

Two ways to make content smaller without dropping it:

Compression removes what carries no information: boilerplate, repeated headers, unused fields, navigation text from web pages, long stack traces cut to the relevant frame.
Summarisation (often called compaction) replaces old turns with a short written account, keeping recent turns in full.

A good compaction summary keeps: the goal, constraints, decisions made, actions taken with their ids, what was learned, and what is still to do.

Why it matters

Long tasks cannot keep everything, and what is kept decides whether the agent continues sensibly.

Compression is nearly free and loses nothing useful: do it always.
Summarisation loses detail, so do it late and carefully.
Compact too early and the agent forgets what it just did. Compact too late and one turn overflows.

In the real world: After compaction dropped a 'do not email the customer' instruction and the agent emailed them, a team added a DO NOT heading to its summary prompt; the mistake did not recur.

Visual example

Compaction

Compression first, because it loses nothing. Summarisation last.

Keeping a long task inside its budget

Compress at sourceloses nothingTools return only needed fields
Clear stale resultsReplaced outputs removed
Offload bulkSaved elsewhere; a reference stays
Summarise old turnsloses detailPast a threshold
Keep recent turnsLast steps verbatim

COMPACT_PROMPT = """You are compacting an agent's working history so it can continue the task.
Write a summary with these headings, and nothing else:

GOAL: the user's request and every constraint, verbatim where short
DONE: actions taken, each with its ids and amounts
LEARNED: facts established, with the source of each
OPEN: what remains, in order
DO NOT: anything the user or a tool has ruled out

Keep all ids, amounts, dates, file paths and error messages exactly as written."""

Summaries of summaries. Each pass loses detail, and after three the agent is working from a rumour. Keep the raw history in storage and summarise from that, not from the last summary.

Ordering, caching and keeping the goal in view

What it is

Where content sits in the context matters as much as what is there:

Stable content first: system prompt and tool definitions. Prompt caching lets a provider store this unchanged start and charge a fraction for re-reading it.
Changing content after: history, results.
Most important content last: the current goal, the plan and the newest result. Models attend most reliably to the beginning and the end.

Why it matters

Order affects both cost and behaviour.

A cache only applies to an identical prefix. One changed character early on, such as a timestamp in the system prompt, invalidates everything after it.
With caching, the fixed part of every turn costs around a tenth as much.
In a long context the original request drifts far from the end. Restating the goal each turn keeps the agent on it.

In the real world: A team finds its cache hit rate is zero because a request id was interpolated into the system prompt; moving it to the last message cuts input cost by more than half.

Putting the current date and time at the top of the system prompt. The prefix then changes every minute and the cache never hits. Put changing values at the end.

Practise it: Design an AI Coding Assistant

Context failure modes

What it is

Four named ways a context goes wrong:

Context poisoning: a wrong fact, often a hallucination, gets into the context and is then treated as true on every later turn.
Context distraction: so much history that the model follows the history's pattern instead of reasoning afresh.
Context confusion: irrelevant material, such as tools or documents that do not apply, pulls the answer off course.
Context clash: two parts of the context contradict each other, such as an old instruction and a new one.

Why it matters

Each has a different cure, so naming the failure is half the fix.

Poisoning: verify facts before they are written to state or memory.
Distraction: summarise and trim.
Confusion: select tools and documents per turn.
Clash: remove the superseded content instead of adding a correction after it.

In the real world: An agent invented an order id at step three and reused it for ten steps; validating ids against tool results before they enter the state stopped the poison at the source.

Visual example

Failures

Diagnose which one before changing the prompt.

Four ways a context goes wrong

failure	what happened	cure
Poisoning	A made-up fact entered and stayed	Validate before storing
Distraction	History so long it is imitated	Summarise and trim
Confusion	Irrelevant tools or text misled it	Select per turn
Clash	Two parts disagree	Remove the superseded part

Correcting an error by appending 'ignore the earlier value'. Both values are now in the context and the model may use either. Remove or replace the wrong one.

Interview questions: context engineering

What it is

The questions interviewers ask about context engineering, from the first screening call to the senior round. Cover the answer, say yours aloud, then compare.

Why it matters

Reading an idea and explaining it are different skills.

Each answer below is the two or three sentences an interviewer is listening for.
The reasoning behind each one is in the cards above. If an answer surprises you, go back to that card.

In the real world: A candidate asked why their agent got worse after turn fifteen answers 'the goal had scrolled eight thousand tokens up'; the interviewer writes 'has shipped one'.

Q: What is context engineering?

Deciding what the model reads on each turn: instructions, tools, memory, retrieved documents, history and intermediate results, selected, ordered and sized by code. Prompt engineering is the wording of one part of it.

Q: Why does context grow in an agent, and why does it matter?

Each step adds a tool request and result, and the whole history is resent every turn. Cost therefore rises faster than step count, and long contexts also reduce accuracy.

Q: How do you manage a long-running agent's context?

Trim tool results at the source, clear stale ones, store bulky outputs outside the context, summarise old turns while keeping recent ones verbatim, and restate the goal and plan at the end each turn.

Q: What is token budgeting?

Giving each category of content a ceiling within a working budget well under the model's maximum, reserving room for the reply, and defining what is dropped or summarised when a ceiling is exceeded.

Q: How does prompt caching affect design?

Caches match an identical prefix, so stable content goes first and changing content last. Do not vary the system prompt or tool list per turn, and never put timestamps at the top.

Q: What should a compaction summary keep?

The goal and constraints, decisions, actions with ids and amounts, facts learned with sources, what is still open, and anything ruled out.

Q: What is context poisoning?

A false fact entering the context and being relied on from then on. Prevent it by validating values against tool results before they are written to state or memory.

Q: Bigger context window or better selection?

Selection. A larger window raises the ceiling but not accuracy or cost efficiency; models use a short relevant context better than a long complete one.

Answering every context question with 'use a model with a bigger window'.

Exercise: put an agent's context on a budget

What it is

A short build that uses this module's ideas. Write it yourself before looking anything up; the checks at the end tell you whether it works.

Why it matters

An idea you have only read fades in a week.

Building it once, even in fifty lines, is what makes it yours.
Interviewers can tell the difference between someone who has read about a loop and someone who has debugged one.

In the real world: A platform team runs this as a standing review: every agent must show its per-category token table before it is allowed more model budget.

Your task. Measure where your agent's tokens go, then cut them with trimming, selection and compaction, and show the saving.

Steps

Take the from-scratch agent and add a function that reports input tokens per category for each turn.
Make one tool return a large payload (a 200-line fake order history) and run a ten-step task. Record total input tokens.
Trim that tool to return only the lines relevant to the request, and run again.
Add a compaction step that summarises all but the last four turns once the history passes a threshold, using the structured summary prompt.
Append a goal-and-plan reminder as the final message of every turn.
Check your provider's usage figures to confirm the stable prefix is being read from cache.

You are done when

You have a before-and-after table of total input tokens and cost.
The task still succeeds after compaction, with ids and amounts intact.
Cache-read tokens are above zero from the second turn.
You can name the largest remaining category and what you would do about it.

Stretch. Deliberately poison the context with a wrong order id at step two and add the validation that prevents it being stored.