# Diagnose misbehaviour: four causes by timing, the cheapest fix first, and where the fix lives (/docs/agent/diagnose-misbehaviour) --- type: Procedure title: "Diagnose misbehaviour: four causes by timing, the cheapest fix first, and where the fix lives" description: "When a worker or an assistant disappoints, the procedure from The AI Agent Factory: tell four causes apart by when the symptom appeared, check the cheap things first, turn a reaction into an instruction, file the fix as a rule, a reference or a procedure, measure the before, and say what the system does without overstating it." status: stable order: 13 generated: { by: "process:claude-code", at: 2026-09-09T06:45:00Z } verified: - { by: "human:rrizwan98", at: 2026-09-09T07:47:26Z } sources: - id: af-workflow title: "Workflow Design & Diagnosis: A Crash Course. The AI Agent Factory, Panaversity, 2026" resource: https://agentfactory.panaversity.org/docs/workflow-design-diagnosis-crash-course ksor: audience: [public] owner: human:rrizwan98 approval: { by: "human:rrizwan98", at: 2026-09-09T07:47:26Z } trust_tier: human-reviewed build_id: sha256:4d4bff09bd0a1dcff0a0a50f20f7ef514fc471e2b1f9e81e0e7b6150b93c0793 dirty: true --- A worker built from this room will disappoint you at some point, and so will the coding agent that built it. When output disappoints, most people do one of two unproductive things: give up and conclude the tool cannot do this, or thrash, changing things at random until something works. Both skip the only question that narrows it down: **when did the symptom first appear?** [^af-workflow] This page is the book's procedure for that moment, in the order it runs: diagnose by timing, fix the cheapest thing first, make the fix stick, measure it, and explain the result to somebody who was not in the room. ## The loop, and the cycle that fails quietly Building with an AI is a loop, not a vending machine: ideate produces options, prototype makes one concrete enough to react to, feedback exposes what is wrong with it, refine fixes that, then round again. Run it inside one project so the context, constraints and prior decisions stay stable; run it across scattered chats and every round restarts from a slightly different understanding, which is how you end up with a pile of drafts instead of a solution. [^af-workflow] The book's worked loop is three cycles of an internal dashboard, no code written by hand. Cycle 1 builds five charts, and works. Cycle 2 adds a date filter and a totals row: the filter works and the totals are wrong, not obviously, the right order of magnitude, formatted correctly, sitting in a row that looks exactly like a totals row should. One analyst notices because she happened to know roughly what one number should be. Nothing broke. The totals were written rather than computed, and a written number is a plausible number. Cycle 3 asks for the brand palette and a print layout, and it works first time. [^af-workflow] The contrast is the lesson of the whole loop. Both were one-sentence requests. Cycle 3 was a **description problem**, and describing it better solved it. Cycle 2 was a **feature problem**, and no amount of describing would have solved it, because the thing asked for was not the kind of thing prose produces. Telling those two apart is the skill, and the realistic failure in a loop is the one that passes review, because the output was fluent, formatted and confident. The catch has to be structural. [^af-workflow] In this room the structural catch is the tool body: a number the answer depends on is computed in code and returned in the tool result, so a written total cannot reach the user. That is rule 9 of the skill and the reason [tests first](tests-first.md) tests the plain function before the model ever sees it. ## Four causes, told apart by timing Four causes produce bad output that looks broadly similar on the page. Timing narrows it down fast, and timing is free information you already have. One rule before the four, because it decides how you use them: **timing tells you where to look first. It does not tell you what the cause is.** Each signal gives a leading hypothesis and a cheap way to test it. [^af-workflow] | When the symptom appeared | First hypothesis | Cheap confirmation | Fix if confirmed | | --- | --- | --- | --- | | Wrong from the first answer | Under-specification | Re-read the prompt. Is the missing thing actually in it? | Add what it left out | | Started fine, then degraded | Context overload | Restate the instruction in a fresh session. Does it hold? | Restart or summarise | | One repeatable error type | Wrong feature or model | Run the step once with the right feature or a stronger tier | Change the feature or the tier | | Used to work, now does not | Stale configuration | Open the instruction or knowledge source and check its date | Maintenance on the configuration | In the book's words: [^af-workflow] - **Under-specification** shows up from the very first response. The output was never right, because the prompt never carried what it needed. The most common cause and the cheapest to fix. - **Context overload** shows up partway through a session that started fine. The conversation approaches its limit, earlier content gets compressed to make room, and early instructions lose force. A better prompt does not help, because the problem is not the prompt you just wrote. - **Wrong feature or model** shows up as a specific, repeatable error type rather than general poorness. Numbers subtly off means a calculation was asked for in prose. Persistently shallow work on a task that needs depth means a tier chosen for speed. - **Stale configuration** shows up as "it used to work". Something the setup depends on has drifted. Nothing errors; output just quietly degrades. Every confirmation in the third column takes under a minute, and each one can come back negative. When it does, you have not wasted the minute; you have eliminated the most likely cause. [^af-workflow] Stale configuration is the one that fails silently. Last quarter's figures sit in the knowledge base and this quarter's report quotes them, formatted perfectly; the model has no way to know what you gave it is out of date. It is the twin of the unstaffed gate on [understand the work](understand-the-work.md): a control set up correctly, decaying while everyone assumes it holds, found by scheduled review rather than by noticing. [^af-workflow] For a worker, the configuration includes the SDK it runs on, which is why the pages in this room that name a version carry a review date and why the skill re-probes the SDK before touching it. ## The sequence runs cheapest first Run these in order, before concluding the task is not a fit: [^af-workflow] 1. **Re-read the prompt against its five components.** The role you asked for, the context the model could not have known, the task as an unambiguous instruction, the constraints, and the output format. A prompt reads fine to the person who wrote it, because the missing piece is sitting in their head. Going slot by slot is what makes an absence visible. 2. **Check the conversation length.** Is context overloaded and in need of a restart or a summary? 3. **Check the feature and the model.** Is this a calculation that needs code execution, or a complex task on a speed tier? 4. **Check the configuration.** Are the instructions, knowledge and skills current? 5. **Only then, question whether the task is a fit at all.** The order is cheapest fix first. Re-reading the prompt costs seconds and resolves the most common cause. Questioning the task is last because it is the most expensive conclusion available: it ends the attempt. Most people invert this; the instinct on a disappointing output is to switch to the most capable model or declare the task impossible, both moves from the bottom of the ladder made before anything at the top was checked. [^af-workflow] Each rung is a hypothesis plus its confirmation, not a guess plus a change. Confirming under-specification costs seconds, so a negative result costs seconds too, and you move down having genuinely ruled something out. [^af-workflow] Step 5 exists and is sometimes right. "I asked it to predict next quarter's exact figure and the number was wrong": no prompt, restart, feature change or configuration audit fixes that, because the task asks for something the tool cannot do. The fix is to the task: ask for a range with stated assumptions, or a model of the drivers you can adjust and re-run. Reached first, step 5 is an excuse. Reached last, it is a conclusion. [^af-workflow] The book's gallery, worth reading as a set because the diagnoses differ and the complaints do not: [^af-workflow] | What you would say | Diagnosis | Fix | | --- | --- | --- | | "The summary keeps missing key points." | Under-specification | Name the criteria for what counts as key | | "It stopped following my format halfway through." | Context overload | Restart from a summary, or persist the format | | "The numbers are subtly wrong." | Wrong feature | Move the calculation to code execution | | "It worked last month, now it's off." | Stale configuration | Audit the instructions and knowledge sources | | "It can't predict next quarter's exact figure." | Expectation mismatch | Reshape the task into one that fits | When you hand the problem to somebody else, say it the other way round: the failure lives in the prompt, the context, the feature choice, or the task itself. That sentence names what they should open. [^af-workflow] ## Turn a reaction into an instruction Every disappointing output is diagnostic data about the setup that produced it, and most of it evaporates because the fix gets made by hand. The first move is converting a reaction into an instruction. A reaction names how the output feels: too generic, not quite right, missed the point. An instruction names what to change so the next output is different. The question that makes the jump: **what specifically would have to be present for this to be right, and which part of the setup controls that?** [^af-workflow] | Reaction | Instruction | Which lever | | --- | --- | --- | | "Too generic" | Name the audience and the one action I want them to take | The prompt | | "Wrong tone" | Add a tone constraint that applies to every draft | The instruction | | "Missed the point" | State the single question the output must answer, up front | The prompt | | "It's using old data" | Replace the source document in the knowledge base | The knowledge | If you cannot name the lever, the critique is still a reaction, and the next attempt will be a guess that looks like a revision. [^af-workflow] ## Rule, reference or procedure Finding the fix is the easy half. The expensive failure is finding it and then losing it: a correction discovered on Monday and left in that conversation is rediscovered next Monday, and again by whoever covers for you. The test for promotion is short: **will this correction be needed again, by me or by someone else?** If yes, it belongs in configuration. Then one more question decides where, because each home changes a different thing: a rule changes behaviour, a reference changes what is known, a procedure changes how the work is done. [^af-workflow] - **Rules go to standing instructions.** Something that should always apply. "Always state the target segment in the first line." - **References go to the knowledge base.** Material every run needs to consult. The brand voice guide, the current product list. - **Procedures go to skills.** A repeatable sequence with steps. "Generate the weekly report in this exact format, in this order." Putting a fix in the wrong home is why some optimisations do not stick. A procedure pasted as a one-line instruction loses its steps and produces a different result each time. Reference material crammed into a standing instruction bloats every prompt whether or not that run needs it. [^af-workflow] For a worker built here, the three homes are concrete: a rule goes into the constitution or the agent's instructions (written under the [prompting guide](../prompting/overview.md), so only what the model cannot work out on its own); a reference goes into the record or the documents the worker reads through a tool; a procedure goes into a tool body or into the skill your coding agent runs. And a rule that must hold regardless of the model goes into a tool body, never only into the prompt. Memory is not the answer here. An assistant can pick up patterns you repeat, but that memory is per user, so the colleague covering for you starts from nothing, and it is best-effort: you cannot point at it, inspect it or hand it to somebody. Configuration is the deliberate, shared, inspectable home. Use both; rely on configuration. [^af-workflow] ## Find the friction before you optimise Friction is easy to live with and hard to see, because you absorb it one session at a time. Three signals point at friction that configuration can remove: [^af-workflow] | Signal | What it looks like | The fix | | --- | --- | --- | | Repetition | You paste or type the same thing every run | Saved context or a standing instruction | | Correction | You fix the same flaw in every output | A configuration change so it stops appearing | | Variance | Different people running the same task get different results | A shared skill or knowledge base | Variance is the one teams miss, because no individual experiences it; it shows up at the reviewer, not at the source. If you cannot see the friction, run the workflow once with the explicit job of writing down every manual step. [^af-workflow] Two moves remove friction, in this order: **consolidate** (steps that can run together should run together, rather than three prompts with you re-explaining context in between), then **promote** (move the repeated pattern into configuration). Consolidation reduces the number of steps; promotion reduces the cost of the steps that remain. [^af-workflow] ## Measure the thing you actually care about There is a step before measuring an improvement that almost everyone skips: **measure the before.** Run the workflow once, unchanged, and record three numbers: how long it took, how many revision rounds it needed, and how many manual steps you performed. Do it before you touch anything; you cannot reconstruct a baseline afterwards, because by then your estimate of the old way will be wrong in the direction that flatters you. [^af-workflow] Time saved is the easiest metric and often not the right one. Pick the metric that matches why the workflow mattered: [^af-workflow] | If the workflow is | Optimise for | Because | | --- | --- | --- | | An internal draft | Time | Speed is the point, and small errors are cheap | | A customer-facing report | Consistency | Varying formats erode trust faster than delay | | A compliance or finance output | Accuracy | One wrong figure costs more than an hour saved | | Work done by several people | Variance | The cost lives between people, not within a person | Naming the metric up front also tells you when to stop: once it is good enough, further tuning becomes its own kind of friction. Two habits make the change survive contact with other people. Run the new workflow in parallel with the old one for a few cycles, which buys evidence, a fallback, and the agreement of the people whose work changed. And be able to say what it costs: the usage attributable to the workflow plus the human time still in it, which is the gates you deliberately kept, against the baseline multiplied by how often it runs and by how many people run it. Then name the risk that survives your gates, because a business case that hid one stops being believed the first time something goes wrong. [^af-workflow] This is the baseline the [count](../method/count-your-own-workflow.md) promised and the number the [evals](evals-before-shipping.md) are measured against. A worker with no "before" has no improvement to report, only a belief. ## When a bad output gets out Everything above reduces the chance of a bad output reaching somebody. Nothing removes it. A gate gets skipped on a busy Friday, a configuration goes stale between reviews, a correctly classified step still produces something wrong. The quality of the design is then judged by what happens in the next hour rather than by the map. What you do in that hour, stop the spread, record the facts, report through your organisation's path, is the first-hour procedure on [govern it](govern-it.md). What the person who designed the workflow decides in advance, so that procedure has something to act on, is four things: [^af-workflow] 1. **How it gets stopped.** The fastest way to halt this workflow, who can do it, and whether they need anyone's permission. If the honest answer is "we would have to ask the person who set it up", that is the first thing to fix. 2. **What is already out.** Not whether one output went wrong, but how many outputs this affected and where they went. A stale configuration produces every result since it went stale. 3. **Who is told, and by when.** Decided before it happens, because in the moment the incentive to delay is strong and a late disclosure costs more than an early one. 4. **What changes in the map.** Which of the four causes was it; was a step misclassified, or classified correctly and the gate not staffed. Make the change with a date, so the fix persists past the week everyone was paying attention. A team that can say what it does when the workflow is wrong is a team that gets permission to run the workflow. [^af-workflow] In this room, decision 1 is the kill switch in the [harness](loop-and-harness.md), decision 2 is the trace the harness keeps, and decision 4 is the eval case you add for the failure. ## Say what it does, then name the gate Integrating a worker into a team means describing it to people who did not build it. Credibility comes from accuracy, which means stating the limits as clearly as the value. Three phrases quietly overstate and all three are common: "fully automated" (almost never true, and the first visible error exposes it), "AI handles X" (collapses the human gate out of the sentence), and "it's basically as good as a person at Y" (a standard the tool will eventually miss in front of somebody who remembers you said it). The repair is the same every time: **state what the tool does, then name the human checkpoint.** Two clauses. A third is worth adding when the audience is deciding whether to trust the workflow at all: say what it refuses. [^af-workflow] The book describes one contract workflow three ways, and the gate does not change; what changes is the detail each audience needs. The technical stakeholder gets the mechanism and the known failure. The executive gets the result in the units they care about, turnaround time, not product names. The risk function gets one sentence about the control and nothing else. **Stakeholders trust an AI workflow more, not less, when the human checkpoints are explicit.** The instinct to minimise the human involvement has the opposite effect: an unexplained gap between "the AI does it" and what happens when the AI is wrong is exactly where a risk function stops listening. [^af-workflow] This is why this record reports the delivery run as seven AI steps and three human decisions, and never as "automated". The honest sentence naming what the tool misses can only be written after diagnosis, which is why this page sits where it does. ## The sentence to carry The output is not the verdict. Judge the step by what it costs and who answers for it, and judge a failure by when it started. [^af-workflow] When the worker is diagnosed and the fix is filed where it belongs, continue to [FastAPI, Docker, Vercel](fastapi-docker-vercel.md) to put it behind a URL, or to [evals before shipping](evals-before-shipping.md) if the fix changed behaviour, because every fix that changed behaviour is a new case. [^af-workflow]: Workflow Design & Diagnosis: A Crash Course. The AI Agent Factory, Panaversity, 2026.