1. Clarifying questions to ask
- How many tenants, how different are their policies and tools?
- Which channels: chat, email, voice?
- What actions may agents take: refunds, account changes?
- How do human staff fit in: approval, take-over, review?
- What are the latency and cost targets per conversation?
2. Requirements to state back
- Resolve common requests per domain; hand over the rest with a summary.
- Strict tenant isolation of data, memory, prompts, policies and limits.
- First response within two seconds; actions within policy and audited.
- Must never cross tenants or customers, act outside a tenant's policy, or pay twice.
3. High-level design
- A gateway authenticates the end customer and attaches tenant and user identity.
- A router (small model) picks a specialist: billing, orders, technical, account, or human.
- Each specialist is an agent with its own prompt and a small tool set, loaded from the tenant's configuration.
- Hand-offs between specialists carry a typed package: goal, facts, actions taken, what remains. Allowed hand-offs and a cap are enforced in code.
- One shared tool layer for all specialists: validation, authorisation as the customer, the tenant's policy engine, idempotency, audit.
- Per-conversation state checkpointed; a human console for approvals, take-over and review.
- Per-tenant queues, limits, budgets, evaluation sets and dashboards.
4. Where to go deep
- Tenant isolation: tenant id on every query, index, cache key, memory namespace and trace; row-level security; per-tenant encryption keys where required; a cross-tenant leak test in the release gate.
- Why specialists: distinct tools and policies keep each context small and each tool list short; a new domain is a new agent, not a longer prompt. But one router plus one tool layer keeps control central.
- Hand-offs: one owner at a time; the package carries actions taken to prevent repeats; a ledger and idempotency keys as backstop; at most three hand-offs then a human.
- Configuration as data: each tenant's policies, caps, approval thresholds and tone are versioned configuration, evaluated against that tenant's test set before release.
- Scale: stateless async workers; per-tenant concurrency and token budgets to prevent noisy neighbours; a shared rate limiter against the provider quota.
5. Trade-offs to name
- Specialist agents versus one agent with tool routing: clearer contexts against more hand-offs to get wrong.
- A supervisor in the middle versus direct hand-offs: an overview and an extra hop against speed and possible ping-pong.
- Per-tenant customisation versus a shared, well-tested core.
- Autonomy level per tenant: a new tenant starts in suggest or approval mode and earns more.
6. Follow-up questions, and how to answer
Q: Billing refunds a fee, hands to orders, and orders refunds it again.
The hand-off package lists actions taken, the shared action ledger is checked before any write, and the refund's idempotency key is derived from the conversation and the action, so the payment service performs it once.
Q: One tenant runs a sale and sends ten times its usual traffic.
Per-tenant concurrency and token-per-minute limits, separate queues or fair scheduling, and that tenant's budget. Others are unaffected; that tenant sees queueing and, if needed, a degraded mode.
Q: A tenant wants to change its refund policy today.
Policy is versioned configuration in the policy engine, not prompt text. The change runs against that tenant's evaluation set, then a canary on their traffic. No code deploy.
Q: Why not one big agent for everything?
With four domains and fifty tools per tenant, one context would carry every tool and policy, selection accuracy would fall and cost rise. But I would keep it to one router, one tool layer and one state store, so only the reasoning is specialised.
Q: How do you evaluate such a platform?
Per tenant and per specialist: end-state checks in a sandboxed copy of the tenant's systems, hand-off cases, refusal and adversarial cases, a cross-tenant leak test; online, resolution, escalation and reversal rates by tenant.