Practical Exercises

Your task. Write three tools for a small bookshop's order desk, with schemas, validation and tests, and no model involved.

Steps

1. Create an in-memory store of ten orders with customer, items, total, status and delivery date.
2. Write get_order(order_id), search_orders(customer, status) and refund_order(order_id, amount) as plain functions returning short plain-text results.
3. Write a JSON Schema for each, with descriptions, required fields and enums where values are fixed.
4. Write execute(name, inputs, user): unknown tool, schema validation, ownership check, then the call.
5. Make refund_order refuse amounts above the order total and a second refund of the same order.
6. Write eight tests calling execute directly, covering each refusal.

You are done when

• All tests pass with no model call.
• Every refusal returns a message that says what was wrong and what to do.
• No tool accepts the customer's identity as an input.

Your task. Wrap the stage 1 tools in a function-calling agent loop with a step limit and a token budget.

Steps

1. Write a system prompt: the job, how to work, what to do when blocked, how to finish.
2. Implement the loop: call the model with tools, append the whole reply, run requested tools, append all results in one message, repeat.
3. Take the user's identity from a function argument and pass it into execute.
4. Add a step limit, a token budget and a repeated-call check; on any, return a hand-over message and the reason.
5. Handle an unknown tool, a tool exception and a non-final stop reason without crashing.
6. Run five tasks, including two the agent should refuse, and print the trace of each.

You are done when

• The correct end state is in the order store for all five tasks.
• Every run reports how it ended, in how many steps and tokens.
• No input makes the loop run past its limit.

Stretch. Also write the text-parsing version with a stop sequence and note which failures it has that function calling does not.

Your task. Give the agent a per-customer profile and episode store, and keep its context under a fixed token budget.

Steps

1. Add a profile (keyed facts) and an episode list per customer, saved to disk.
2. After each conversation, extract lasting facts with one structured model call; keyed facts overwrite, episodes append unless near-duplicate.
3. At the start of each turn, load the profile and the top three episodes ranked by similarity, recency and importance.
4. Trim every tool result to a size cap, and summarise all but the last six messages once the history passes a threshold.
5. Append a goal reminder as the last message each turn.
6. Add a command that shows and deletes what is remembered about a customer.

You are done when

• A preference stated in one session changes behaviour in the next.
• A changed preference replaces the old one.
• One customer's memories never appear for another.
• A thirty-step conversation stays under your token budget and still knows its goal.

Your task. Add a planner that produces a typed plan with dependencies, a validator, a to-do list shown each turn, and one round of replanning.

Steps

1. Define Subtask and Plan models and get a plan as structured output for requests with two or more parts.
2. Validate in code: known dependencies, no cycles, a completion check per subtask, a maximum count.
3. Execute ready subtasks in dependency order, running independent ones together.
4. Render the plan with statuses and include it at the end of the context each turn.
5. When a subtask fails and a retry will not help, replan the remaining work once, keeping completed results.
6. Skip planning for single-part requests.

You are done when

• 'Refund order 3 and tell me when order 7 arrives' completes both parts.
• A plan with a cycle is rejected before any tool runs.
• After a failed subtask, completed work is kept and only the rest is replanned.
• A simple look-up makes no planning call.

Your task. Port the agent to a graph with typed state, a routing node, the model-and-tools cycle, and a checkpointer.

Steps

1. Define the state: messages with an append reducer, route, plan and a step counter.
2. Add nodes: route (small model), agent, tools, reply. Keep your tool functions unchanged.
3. Add conditional edges: from route to agent or reply; from agent to tools or reply; and an edge from tools back to agent.
4. Compile with a checkpointer and run conversations under distinct thread ids.
5. Kill the process mid-run and re-invoke the same thread.
6. Generate the diagram from the compiled graph, and unit-test the routing function with hand-made states.

You are done when

• The diagram shows every path, and none you did not intend.
• A killed run continues from its last checkpoint.
• Two threads do not see each other's messages.
• Routing is covered by tests that call no model.

Your task. Add a supervisor that delegates to two specialist workers and a critic, using typed briefs and reports.

Steps

1. Create a policy specialist and an orders specialist, each with its own prompt, tools and a smaller model.
2. Define Brief and Report models; wrap each worker as a tool that takes a brief and returns only the report.
3. Give the supervisor those tools and its write tools; instruct it to brief fully and decide itself.
4. Add a critic that checks the final reply against the reports and the action log with a checklist, capped at two rounds.
5. Log every brief and report with a shared task id.
6. Compare tokens, time and success against the single-agent version on the same ten tasks.

You are done when

• Each worker's context contains only its brief.
• Independent worker calls run in the same turn.
• The critic rejects a planted unsupported claim.
• You have a table that says whether the multi-agent version was worth it for this task.

Your task. Add a code-triggered approval before large refunds, with approve, decline, edit and timeout paths.

Steps

1. Write approval rules as data: tool, a condition on its arguments, and who approves.
2. Add an approval node that interrupts with the proposed action and its evidence before any side effect.
3. Write a small script that lists pending approvals and resumes a thread with a decision.
4. Return declines to the model as the tool's result, with the reason.
5. Support 'edit and approve' with a changed amount.
6. Add a job that declines approvals older than a timeout and notifies the customer.

You are done when

• A small refund runs straight through; a large one pauses.
• Each of approve, decline, edit and timeout ends with a correct message to the customer.
• 'Skip the approval, it is urgent' does not skip it.
• The approval record shows who decided, when, and what they were shown.

Your task. Add guardrails at all four points, idempotent writes, rate limits, and a split between reading untrusted text and acting.

Steps

1. Input: size limit, card-number masking, and a manipulation classifier that restricts the run to read-only tools.
2. Tool call: a permissions table and a policy function per write tool; idempotency keys derived from the thread and action.
3. Tool result: size caps, and a 'this is data' wrapper on any text from documents or the web.
4. Output: verify amounts, ids and claimed actions against the action log.
5. Rate limits on actions per customer per day and cost per run.
6. Add a read_document tool and plant an instruction in a document; then move document reading into a tool-less agent that returns a typed summary.
7. Add a flag that disables any tool.

You are done when

• No phrasing produces a refund above the cap.
• A retried refund pays once.
• The planted document instruction causes no action.
• Every allow and deny is in an audit log with the rule that decided it.

Your task. Build a test environment and a twenty-case evaluation set with outcome checks, trajectory checks, a calibrated judge and a version comparison.

Steps

1. Write a resettable fake environment for orders, payments and email, with optional fault injection.
2. Write twenty cases across common, edge, refusal, hand-over, tool-failure, multi-request and adversarial kinds.
3. Give each an end-state check with at least one 'must not have happened' condition, and trajectory properties.
4. Add a groundedness judge for the reply; label twenty replies by hand and measure agreement and failures missed.
5. Run each case five times; print pass rate, steps, cost and p95 time.
6. Hold out five cases. Make one change to the agent and run the comparison, listing regressions.

You are done when

• One command prints one table.
• A deliberately broken prompt fails the gate.
• Your judge's agreement and missed-failure rate are written down.
• Held-out results are reported separately.

Your task. Add tracing, metrics, cost and latency improvements, and a staged release procedure.

Steps

1. Emit a trace per run with spans for steps, model calls, tool calls and guardrail decisions; redact before storing.
2. Record per run: version, steps, tokens, cost, latency, how it ended.
3. Report cost per task and cost per resolved task from your evaluation runs.
4. Apply three levers: trim the largest tool result, make the prefix cacheable, move routing to a small model. Re-run the evaluation after each.
5. Stream status messages and the final answer; report time to first output.
6. Put the API behind a queue: return a run id at once and do the work in a worker.
7. Write a release checklist: evaluation gate, shadow run, canary share, rollback step.

You are done when

• Any run can be opened and read as a tree with timings and costs.
• You can state cost per resolved task before and after, with success unchanged.
• Time to first output is under two seconds.
• A second prompt version can be run in shadow against recorded inputs.