Capstone Project

Build an agent for Dukaan Direct, a made-up online electronics shop. A customer writes in about an order, and the agent must resolve the request, not merely answer it. In scope:

• return requests: check eligibility against the returns policy, issue a return label, schedule a pickup;
• refunds: full, partial or shipping-only, within limits;
• exchanges: check stock and arrange a replacement;
• questions about policy, warranty and order status;
• hand-over to a human, with a summary, for anything it should not decide.

You build the shop's back end as fakes: orders, payments, inventory, courier, email, and a folder of policy documents.

The domain is chosen because it needs every part of the course at once.

• Policies are documents, so it needs RAG.
• Requests have several parts, so it needs planning.
• Money and stock change, so it needs guardrails, idempotency and human approval.
• A return takes days (label, pickup, inspection, refund), so it needs saved state and resumption.
• Customers return, so it needs memory.
• And it must be shown to work, so it needs evaluation and tracing.

The system must meet the functional, quality and safety requirements below. Write your own acceptance test for each before you build.

Requirements are the contract you will be marked against.

• Each one is stated so that it can be tested.
• The 'must never' list is the most important part: each item needs a control in code, and a test that tries to break it.
• If a requirement seems to conflict with another, that tension is intentional; say how you resolved it in your design document.

The capstone must demonstrably include all ten:

• Tools: task-shaped, typed, with read and write separated.
• Memory: a per-customer profile and episodes, with update and deletion.
• Planning: decomposition and tracking for multi-part requests.
• RAG: retrieval over the policy documents, with grounding checks.
• State: checkpointed runs that pause and resume.
• Human-in-the-loop: code-triggered approval, and hand-over.
• Guardrails: at input, tool call, tool result and output; plus limits.
• Observability: traces, and metrics per run.
• Evaluation: a test environment, a case set, and a results table.
• Error handling: retries, idempotency, partial failure and honest reporting.

Each capability maps to a module, and the mapping is your reading list when you are stuck.

• The point is integration: these parts constrain each other. Approval needs saved state; idempotency needs stable keys from state; evaluation needs a test environment that the tools run against.
• You may use a framework or none. Either way you must be able to show the raw request sent to the model.

The brief does not tell you how to build it. These are the decisions, each with the question to answer in your design document. There is more than one defensible answer to every one.

This is what makes the capstone a design exercise.

• A reviewer does not mark the choice; they mark whether you saw the alternatives and can justify yours with evidence.
• Several of these interact: your answer on state constrains your answer on approvals; your answer on shape constrains your evaluation.
• Write each decision as: the options considered, the one chosen, the reason, and what would make you change it.

A suggested order of work over four to six weeks of evenings. Each milestone leaves you with something that runs and is tested.

The order is chosen so that nothing is built on an untested foundation.

• Fakes and tools first: they need no model and catch most design mistakes.
• Evaluation early, at milestone 3, so every later change is measured.
• Safety before polish: guardrails and approval come before cost and latency work.
• If you run out of time, stopping after any milestone still leaves a coherent project.

At least thirty cases, each run five times, covering:

• eight common requests across return, refund, exchange and questions;
• four edge cases: the last day of the returns window, an amount exactly at a cap, a partially shipped order;
• four refusals: outside the window, already refunded, not the customer's order;
• three multi-part requests;
• three long-running returns, resumed after a simulated delay;
• four tool failures: a timeout after success, a courier outage, an email failure after a booking;
• four adversarial cases: a false authority claim, an instruction inside an attached file, a request for another customer's data, a prompt-extraction attempt;
• at least five held out from tuning.

The set is the project's evidence.

• End-state checks on the fake systems make 'resolved' objective.
• Each 'must never' has at least one adversarial case that must pass on every run, not most.
• Fault-injection cases prove the error handling, which nothing else does.
• A calibrated judge is needed only for what code cannot check: whether the reply is grounded in the policy text and complete.

Hand in six things:

• The code, runnable with one command against the fake back end.
• A design document of four pages or fewer: requirements, architecture diagram, the twelve decisions with alternatives, and the threat model.
• The evaluation set and harness, with a command that prints the results table.
• A results table: versions, pass rates, cost per task, cost per resolved task, latency, and what changed between versions.
• Three annotated traces: a clean success, an approval that was declined, and a failure you could not fix, with your diagnosis.
• A ten-minute demonstration: one multi-part request end to end, one attack that fails, and one crash that resumes.

Each deliverable answers a question a real reviewer asks.

• The design document: did you choose, or drift?
• The results table: does it work, and how do you know?
• The annotated failure: can you debug, and are you honest about limits?
• The demonstration: can you show it surviving the things that break agents?

A rubric of six dimensions. Score yourself honestly before asking anyone else to.

The weights say what matters.

• Safety and reliability together are worth more than functionality, because a feature that works unsafely is not finished.
• Evidence is its own dimension: a claim with no measurement scores nothing.
• Complexity earns no marks. A simpler system that meets every requirement outscores an elaborate one that meets most.

The failures seen most often, in rough order:

• No evaluation until the end, so regressions pile up unseen.
• Limits in the prompt, so the first adversarial test passes a refund above the cap.
• A reply that claims an action the ledger does not show.
• State in memory, so the multi-day return cannot be demonstrated.
• A duplicate refund on a retried timeout.
• Policy answers from the model's general knowledge, not the documents.
• Too many agents, with hand-offs that lose the customer's second request.
• The attachment attack works, because the model that reads the file also holds the refund tool.

Every one of these was the subject of a module, and each has a mechanical fix.

• They are common because each is invisible on the happy path.
• Read this list again at milestones 4, 8 and 11, and test for each deliberately.
• If you fix all eight, you have built something better than many systems in production.