# Design the experience: trust, autonomy, recovery, and the machine surface (/docs/agent/design-the-experience) --- type: Procedure title: "Design the experience: trust, autonomy, recovery, and the machine surface" description: "How a person comes to trust a worker that acts on its own, from The AI Agent Factory: the trust formula, first contact, progressive transparency, the five-stop autonomy dial, intent preview, honest waiting, repair and redress, the machine surface other agents parse, the surface as a safety control, the experience scorecard, the anti-patterns, and the eleven-section brief; plus what a test build from this record showed about feeling natural." status: stable order: 6 generated: { by: "process:claude-code", at: 2026-09-09T06:45:00Z } verified: - { by: "human:rrizwan98", at: 2026-09-09T07:47:26Z } sources: - id: af-experience title: "Designing Agent Experiences: A Crash Course. The AI Agent Factory, Panaversity, 2026" resource: https://agentfactory.panaversity.org/docs/designing-agent-experiences-crash-course - id: raza-test-build title: "Owner's review of a support agent built from this record as a test, 9 September 2026 (attestation)" resource: "attestation: Raza Rizwan, owner; review conducted with process:claude-code on 2026-09-09 of a test build made by a coding agent from this record and its skill; not client work" ksor: audience: [public] owner: human:rrizwan98 approval: { by: "human:rrizwan98", at: 2026-09-09T07:47:26Z } trust_tier: human-reviewed build_id: sha256:4d4bff09bd0a1dcff0a0a50f20f7ef514fc471e2b1f9e81e0e7b6150b93c0793 dirty: true --- Everything before this page decided what the worker does and who owns each step. This page is about the thin, decisive layer where a person meets the worker and decides whether to trust it. When software only responded to commands, design meant arranging buttons so a human could drive. When software acts on its own, the human is delegating, and delegation is a relationship. The craft is no longer "make the button findable"; it is **make the machine's judgement legible, its autonomy adjustable, and its mistakes survivable.** [^af-experience] The book's one-line thesis, kept whole because the room is built on it: an agentic product has two users at once, a human who must trust it and other agents who must parse it, and the job is to design a surface that serves both without betraying either. [^af-experience] ## The shift: intent, trust, recovery Computing has had three ways of talking to a machine. In batch computing you specified a whole workflow up front and waited. In command computing you drove step by step, and the burden of knowing how sat on you. In the third paradigm you state the outcome and the agent chooses the steps: the burden of "how" moves to the machine. When you no longer design the steps, you design three things instead: how the person expresses **intent**, how they come to **trust** steps they did not choose, and how they **recover** when the machine chooses wrong. [^af-experience] The interface does not vanish; it moves. Every autonomous agent creates three interfaces that did not exist before: a configuration surface (how do you teach it your preferences), a monitoring surface (how do you see what it is doing), and an intervention surface (how do you step in when it goes wrong). The centre of gravity moves from translating a task into a widget toward shaping a system of intent. [^af-experience] ## Trust is earned, never assumed You cannot design trust directly. You design its inputs, and trust is what accumulates: [^af-experience] > Trust = reliability shown over time × transparency you can check × control > you can feel × mistakes you can undo. It is a product, not a sum: a zero in any term zeroes the whole thing. A reliable black box earns no trust. A transparent agent you cannot steer earns no trust. An agent you can steer but cannot undo earns no trust. And the first move is the humblest: let the agent be visibly uncertain. An agent that hides its doubt and is confidently wrong destroys more trust than one that says "I am not sure about this part, take a look." [^af-experience] The opposite failure is just as real. Once an agent has been right a hundred times, the person stops checking, and stops noticing when it drifts. This is automation complacency, and design guards against it too: keep uncertainty visible even when confidence is usually high, never let genuinely high-stakes work climb to full autonomy, and surface drift at the fleet level. Calibrated trust, neither too little nor too much, is the target. [^af-experience] ## First contact On day one there is no track record. Three moves make first contact honest: [^af-experience] - **Reset expectations at every jump.** When a surface gains the power to act, say so plainly: "I can now do this for you, not just tell you about it. Here is what that means." The most dangerous moment is the one where capability changed and the mental model did not. - **Borrow trust by lowering stakes.** Start at the lowest autonomy stop, show the plan before acting, make the first task small and cleanly reversible. Day-one trust is earned by making the first mistakes cheap, not by promising there will be none. - **Be honest about being new.** "I have not done this with you before, so I will check in more often at first." And the surface must work for everyone. Treat WCAG 2.2 as the floor, then add the criteria an agentic surface specifically needs, written as testable acceptance criteria: status and progress announced to a screen reader, not conveyed by motion alone; the plan, the undo and the path to a human reachable by keyboard; confidence never carried by colour alone; the person can pause, resume or cancel long-running work; notifications adjustable in quantity and intensity; every explanation with a plain-language version. If a criterion cannot be pinned to a specific layer or control, it is an intention, not a requirement. [^af-experience] ## Show who carries what An agent takes three kinds of weight off a person: the cognitive load of analysis and decision, the creative load of drafting, and the logistical load of steps and coordination. The design mistake is to move the weight silently. When the division of labour is invisible you get what the book calls agentic sludge: the person cannot tell what the agent did versus what they still owe, so they redo the agent's work "just to be safe", and the promised time saving evaporates. The rule: **the division of labour must be visible and adjustable.** At any moment the person should be able to see what they handled, what the worker handled, and what is waiting on them, and move the line. [^af-experience] ## Progressive transparency Show nothing and the agent is a black box. Show everything, every token of reasoning and every tool call, and you bury the person in noise. Both extremes destroy trust. The answer is four layers, each a tap deeper than the last: [^af-experience] 1. **The outcome,** one plain line: what the agent did. All most people read, most of the time. 2. **The plan:** the ordered steps it took. 3. **The why:** its rationale, with an honest confidence signal. Not a fake percentage; a real "high / low / unsure" the agent means. A precise-looking "73%" implies a calibration the model does not have, so "unsure" tells the truth where the number only performs it. 4. **The evidence:** the sources, the tool calls, the full trace. For audit, debugging, and the moment something looks wrong. Small concrete surfaces do the heavy lifting: a provenance chip ("based on 3 of your files"), an uncertainty marker on the shaky part, a link to the trace. You are answering the human's real question: should I trust this, and where should I look first if I don't? [^af-experience] ## The autonomy dial Autonomy is not a switch. It is a dial, and the person, not you, holds it. You ship it turned low and let it rise as the agent proves itself, the way a manager gives a new hire more rope after they have earned it. Five stops: [^af-experience] | Stop | The agent | Regime | | --- | --- | --- | | 1 Suggest | Proposes, you act | Human in the loop | | 2 Confirm | Asks before each step | Human in the loop | | 3 Act in limits | Acts inside a budget, asks past it | Human on the loop | | 4 Act and report | Acts, then reports back | Human on the loop | | 5 Autonomous | Runs alone, you audit | Human on the loop | Two ideas make it safe. First, the split between human in the loop (the agent stops and waits for approval) and human on the loop (the agent acts and you supervise): low-stakes, reversible, well-proven work earns its way on the loop; **high-stakes or irreversible work stays in the loop no matter how reliable the agent gets.** Second, safe defaults with per-task consent: a new agent starts at "suggest", and every step up is a choice the person makes on purpose. [^af-experience] In this room the dial is `needs_approval` on a tool: stop 2 is a tool that always pauses, stop 3 is a tool that pauses above a threshold computed in its body, and a tool pinned in the loop forever is one whose approval can never be turned off. The book's own worked example is a refund worker at stop 3, acting up to a limit and stopping for anything above it, with an undo for 24 hours on every refund. [^af-experience] ## Intent preview The cheapest mistake to prevent is the one the agent has not made yet. Before it acts, especially on anything irreversible, show the plan and let the person edit it: "Here is what I am about to do: 1, 2, 3. Change anything?" This single pattern prevents more regret than any amount of after-the-fact explanation. Two rules make it work: **preview scales with stakes** (one internal draft needs a quiet "sending now, undo?"; emailing 500 customers or moving money needs the full plan and an explicit confirm), and **the plan is editable mid-flight**, not just approvable. A plan you can only accept or reject is a wall. A plan you can adjust is a partnership. [^af-experience] This is the plan gate the [worked example](worked-example.md) keeps as its fourth step. ## Design the wait Agentic work breaks the synchronous rhythm: you set an intent, disconnect, and come back to work in progress. Four surfaces make that humane: intent capture clear enough to run without you; progress you can glance at in three seconds; nudge, don't notify, interrupting only at real decision points; and a review-and-refine surface for the return. The rule for interruptions: earn the interruption. An agent that pings you for every step is worse than no agent. [^af-experience] And design the felt experience of waiting. Agents are slow and they cost money, and **silence reads as broken, not busy**. So: honest progress that names the current step ("reading the 40 invoices" beats a spinning circle), a rough expectation up front, visible cost on expensive runs, and a way to check in without interrupting the work. Latency and cost are part of the experience, not backend concerns to hide. [^af-experience] For a text worker behind a URL, that is the streaming path on [build the text agent](build-the-text-agent.md) and a status line that names the tool being called. ## Repair and redress Your agent will act wrongly. Not might; will. A design that pretends otherwise is not optimistic, it is negligent. Recoverability is a first-class surface designed from the start, and four moves turn a bad outcome from a betrayal into a bump: [^af-experience] 1. **Undo,** as close to one click as the task allows. The single strongest trust-builder in the whole course. 2. **A straight apology** and a plain account of what went wrong, no hedging, no blame-shifting to the user. 3. **The corrective action and the next step,** stated: "I reversed the transfer. I have flagged this for review so it does not recur." 4. **A visible path to a human.** Always. Watch two numbers. Escalation frequency has, by one practitioner benchmark, a healthy band around 5 to 15 percent: too low and the agent is guessing when it should ask, too high and it is uselessly timid. Recovery success, the share of escalated or failed tasks that reach a good end, you want above roughly 90 percent. Treat both as starting numbers to calibrate against your own domain, not laws. [^af-experience] When several workers run at once the surface changes shape, from monitoring the work to triaging the exceptions: a fleet view of who is running, blocked or waiting; attention triage that ranks what needs a human now; and drift, not just distress, so that a worker whose escalation rate doubled this week is surfaced before it fails loudly. The honest form of an attention budget is a circuit breaker: when a worker's error or escalation rate crosses a multiple of its own baseline, the surface drops it to a lower autonomy stop or pauses it for review rather than waiting for a person to notice. [^af-experience] ## The machine surface: your worker has robot users Now the half most teams never design. Your worker is used by agents: the ones acting for your users, and the other workers around it. They do not see your layout; they read your structure, and if that structure is hostile the human's agent fails silently and the human blames you. Four things decide whether an agent can succeed with your product: [^af-experience] - **Access:** can the agent prove whose authority it acts under, with a credential that is scoped and revocable? - **Context:** can the model understand what your product means? Clear names, honest descriptions. - **Tools:** are your capabilities machine-readable, typed and discoverable? - **Orchestration:** can agents chain your capability with others safely: predictable contracts, idempotent actions, sane limits? A well-designed connector or MCP server is good agent experience. The skill file that teaches an agent your task and the MCP tool with a clear typed signature are your product's UX for its robot users. Concretely, a machine surface an agent can trust: [^af-experience] - Name tools for the action, plainly: `refund_order`, not `process`. - Keep schemas narrow, typed and validated. - Declare side effects and danger: mark which tools change the world, and make the dangerous ones require confirmation or a policy check. - Return structured, actionable errors: whether the call is retryable, a retry-after hint, a fallback to try. A code with no next step is half an answer. - Make actions idempotent where you can, so a retry never double-charges. - Carry provenance and permission: where the data came from, whose authority the agent acts under, how it is revoked. - Write the docs for the agent: examples, limits, failure modes, retry behaviour. - Test the contract, not just the screen: a human QA pass never exercises the machine surface. Be precise about what a shared protocol gives you. As of 2026, MCP standardises how a tool is discovered and called, how a resource is read, and how access is authenticated. It deliberately leaves orchestration, governance and state to you. The protocol gets an agent through the door; what it may do once inside, and who is watching, is still your design. [^af-experience] This record's own connector is built on that rule: its tools are typed, its search declares when it abstains, and it never carries an action. A tool no longer has to return only text. With MCP Apps, a tool can return a plain text result for every host and also point to an interactive interface that a supporting host renders inside the conversation, in a sandbox. Use one for exploring complex data, configuring many options at once, rich media, real-time monitoring, or multi-step workflows such as approving item by item. The counter-rule holds: **if a plain text answer serves the person, return text.** A widget must earn its place the way a nudge earns an interruption. [^af-experience] ## The surface as a safety control An agent that acts on the world has an attack surface, and a surprising amount of the defence is design. The book maps the shared checklist of agent risks to the surfaces above: [^af-experience] | Risk | Design defence | | --- | --- | | Prompt injection: a page or document quietly tells your agent to do something the user never asked | Visible provenance, and consequential actions routed through an intent preview | | Excessive agency: the agent can do more than the moment calls for | The autonomy dial and policy surfaces; least authority by default | | Misinformation and over-trust: confidently wrong, and believed | Visible uncertainty and provenance; a shaky claim never wears the face of a sure one | | Unbounded consumption: a runaway loop or a denial-of-wallet attack | Cost shown as it burns; spend limits as a policy surface | | Sensitive-data disclosure and permission abuse | Scoped, revocable credentials and consent gates a person can see and withdraw | Two rules fall out. The surface is a safety control, not just a display, so cutting provenance, uncertainty, previews or spend meters "to reduce clutter" removes protection. And every agentic system needs a governance surface the trust surface can point to: [^af-experience] | Governance surface | The question it answers | | --- | --- | | Capability approval | Who can add or widen what this worker may do? | | Permission review | Who signs off on its tool scopes and data access? | | Incident review | Who owns a failure and its postmortem? | | Audit log | Who can inspect its actions, plans, tool calls and approvals? | | Kill switch | Who can pause, disable or roll it back, in one move? | | Drift review | Who investigates a rising escalation or slipping recovery rate? | Those are the organisational controls behind [govern it](govern-it.md); the surface's job is to make each one reachable. ## Measure the experience Correctness evals measure whether the worker's output is right. Experience metrics measure the relationship: whether a human can delegate, trust, steer and recover. A worker can pass every correctness eval and still wear a surface no one can supervise. Track a small scorecard: [^af-experience] | Metric | What it tells you | | --- | --- | | Plan-acceptance rate | Did people understand and approve the plan, or rubber-stamp and reject blindly? | | Intervention rate | How often did a human have to step in? Trend it: falling means trust is being earned | | Recovery success | After a failure or escalation, did the task still reach a good end? | | Over- versus under-trust | Are people accepting bad output, or rejecting good work? | | Notification precision | Of the interruptions you sent, how many were worth it? | | Time saved versus attention spent | Did the agent cut total human burden, or just move it? | Watch trends, not snapshots. And run six tests before you trust a worker in front of people, one per surface: a plan-review test, an over-trust test (feed it a task where the right-looking output is subtly wrong), a recovery test (let it make a reversible mistake on purpose and time the undo), an interruption test, an accessibility pass with keyboard and screen reader only, and a machine-surface contract test (wrong types rejected, dangerous actions gated, errors structured, retries idempotent). Passing them is not proof the worker is correct; it is proof the experience holds when something goes wrong. [^af-experience] ## Anti-patterns Each is one of the principles above, broken: [^af-experience] | Anti-pattern | What it looks like | | --- | --- | | The black box | Acts with no visible reasoning | | Agentic sludge | You re-check everything, so you save no time | | The over-eager agent | High autonomy on day one, no earning it | | The over-trusted agent | High autonomy on work nobody checks | | Notification spam | Pings for every step | | The alarm-fatigued console | Every worker pings; the human tunes out | | False confidence | States shaky guesses as certain fact | | The trap door | An action that cannot be undone | | The confused deputy | The surface cannot tell a user's instruction from an injected one | | The unmeasured surface | It feels elegant, but no one tracks whether it helps | | The mystery-meat API | A machine surface no agent can parse | ## The brief you leave with The deliverable is an Agent Experience Brief for one worker, a page or two, eleven sections, one per decision: the two audiences; first contact; the trust surface; the load map; the autonomy ladder, with the actions pinned in the loop forever; the async plan; the recovery plan with its two metrics; at scale; the machine surface judged on access, context, tools and orchestration; the safety surface; and the scorecard, with the line where correctness evals take over. It is a spec for the experience layer: the place where a non-deterministic worker is given a deterministic, reviewable surface, written once and returned to every time the worker changes. [^af-experience] In this room it sits beside the [spec](write-the-spec.md), and its autonomy ladder and recovery plan become acceptance criteria. ## What a test build from this record showed In September 2026 a coding agent built a customer-support worker from this record and its skill, as a test on a fictional shop, not client work. The owner reviewed it with sixteen questions that were not in its scripted test set. It answered every scripted question correctly, and it answered ordinary public questions naturally: a greeting, delivery to a city, a product it does not sell, a question outside its scope. It failed the natural questions a signed-in customer asks first. "Where is my order?" got "please give me the order ID", every time, because no tool listed the customer's own orders. A message that mentioned another customer's order alongside a public product question was rejected whole, the public half never answered. A boilerplate sentence about support hours was appended to an answer about something else. And when the model referred to a record it had not fetched in that turn, the person got an error instead of an answer. [^raza-test-build] Every one of those is a concept on this page, and the fixes are not looser guards. They are the following rules, which this room now treats as part of the spec for any worker that talks to people: [^raza-test-build] 1. **Every record a person owns gets a tool that lists it,** scoped by the identity the harness supplies, so "where is my order" is answerable before an identifier is typed. A worker runs against its record; a worker with no way to list the record cannot be natural. 2. **Refuse the private part, answer the public part.** One message can carry two questions. The unavailable record gets the generic line; the rest of the message still gets its answer. That is the visible division of labour. 3. **A well-formed question never ends in a hard error.** If the reply cites something it did not retrieve, run one repair turn that says so, then fail honestly with the path to a human. That is repair and redress, applied to the worker's own mistakes. 4. **Appended facts must belong to the question.** Code may add a total or a status the model omitted, only when the question was about that thing. Layer one is one plain line, not a paragraph of reminders. 5. **Test with questions nobody scripted.** A set of unscripted, multi-intent and off-topic questions sits beside the golden set, and the pass rate on it is reported separately. It is the over-trust test and the interruption test from the scorecard, run on text. These rules do not touch the guards that were right: identity from the harness only, the same generic answer for a missing record and another person's record, no fabricated claims of action, sources that must have been retrieved. Those stay. [^raza-test-build] With the brief drafted, continue to [write the spec](write-the-spec.md); its autonomy ladder and recovery plan are acceptance criteria waiting to be written down. [^af-experience]: Designing Agent Experiences: A Crash Course. The AI Agent Factory, Panaversity, 2026. [^raza-test-build]: Owner's review of a support agent built from this record as a test, 9 September 2026. Attestation recorded in this document's sources; the build was a test on fictional data, not client work.