Building an Enterprise Agent Harness from Scratch7/10: Why Does an Agent Demo Well but Fail in Production?

In the previous chapter, we added permission checks to the refund agent. Now let it complete a demonstration: find the duplicate charge, wait for approval, issue the refund, and send a notification. “Completed” appears on screen. It is easy for the audience to believe the system is ready to launch.

I would pause here and give it a few different questions. If the payment interface times out, will it refund the money again? If the manager has not approved the refund, will it execute anyway? If the money is refunded but the notification never arrives, what will it tell the customer?

One successful demo shows that it can solve that particular task. To decide whether to release it, we also need to know why it succeeds, how it fails, and how reliably it handles the same task across repeated runs.

Follow One Run from Beginning to End

Before grading the model, I open the execution record for one refund and follow it through time. Which order did it read? How much did it decide to refund? When did it receive approval? What did the payment interface return? Which refund ID did it finally find? I also record the versions used and the duration and cost of each step. This record across the task is called a trace.

Each run receives a run ID that connects its events. The model's answer may be correct even though its actions exceeded its permissions. The actions may look normal even though the payment system contains the wrong result. The trace tells us where to look.

OpenTelemetry provides a common recording format. The entire refund is a trace; looking up an order, calling the model, and requesting payment are individual spans, each recording its start, end, and result. Think of the trace as the task's timeline and a span as one step on it. Finding the longest span tells us where to start investigating. A common format lets us connect records from different services. Before collecting them, we also handle sensitive content: mask customer information, keep secrets out of logs, and limit who can read approval materials and how long they are retained.

Trace: A structured execution record that connects the model, tools, policies, approvals, state, cost, and result of one run.

Each Test Must Check the Answer, the Path, and the Result

Next, I turn the duplicate charge from Chapter 1 into a repeatable test, or eval case. The input is still “I was charged twice. Please refund the extra charge.” The test environment contains the same order and two charges: the legitimate charge and the extra one.

I first define the expected result: refund the extra charge exactly once and leave the legitimate charge unchanged. Then I check the process: look up the order first, obtain approval before execution, and access only data belonging to the current customer's organization. Tool calls, cost, and elapsed time also have limits.

This connects back to Chapter 5. The same refund may require several safe call attempts, but it must produce only one refund effect. A timeout-recovery test checks whether the agent queries the original request, meets the conditions for a safe retry, and keeps the original action ID and idempotency key. Starting another refund under a new ID fails the test, even if the final answer sounds correct.

Getting the amount right in the final reply passes only the answer check. Finding exactly one correct refund in the payment system passes the result check. Looking up the order, waiting for approval, and handling timeouts as required passes the path check.

Collect refund, purchase-cancellation, and inquiry-email cases into a test set. Every candidate version takes the same set, with separate checks for results, process, permissions, cost, and elapsed time. A serious refund error blocks release. A passing version first receives a small number of real tasks. Once a new production problem is understood, it becomes a fixed test case so the next version does not repeat it.

Outcome eval: A check that the task produced the correct final result in its environment.

Trajectory eval: A check that tools, arguments, approvals, ordering, and policies were used correctly throughout execution.

Reliability Requires Repeated Trials

Run the same refund test repeatedly. The model may choose different steps each time, and a single run will not reveal that variation. I hold the agent, model, prompt, tool, and policy versions fixed, reset the test data to the same starting point, and record why each run passes or fails. For example, the refund record left by the previous run must be reset in the isolated test environment. Otherwise, the next run is no longer facing the same question.

Reports sometimes use pass@k and pass^k. Here, k is the number of runs of the same task; the example below has three. The former asks, “Does it succeed at least once in k attempts?” and helps describe its capability. The latter asks, “Does it succeed in all k consecutive attempts?” and is closer to the reliability needed for high-risk work.

Under pass@3, it succeeds because at least one run passed. Under pass^3, it fails because it did not pass all three. The metrics read the same record but answer different questions.

pass@k: The proportion or probability of succeeding at least once in k runs of the same task.

pass^k: The proportion or probability of succeeding in all k consecutive runs of the same task.

Release Decisions Must Include Severity, Cost, and Latency

The candidate is the new version being considered for release; the baseline is the existing version used for comparison. Both receive the same tasks, environment, and number of repeated runs. This lets us distinguish the effect of a version change from a change in the questions or environment. Any access across organizations or duplicate refund blocks release. A significant drop in reliability also blocks release. Cost or latency above the budget requires human review.

The upper half of the diagram resets the test environment, runs the agent repeatedly, and checks results, paths, permissions, cost, and time together. A failure goes back for a fix and another run. Passing versions move into the lower half for a limited production rollout. New production failures become fixed regression cases and return to the same test entrance.

A model grader is another model asked to grade work against a rubric. I regularly have people and the grader inspect the same results to see whether their judgments agree. It can assess whether an explanation is clear or the tone is appropriate. Whether a refund ID exists and has the correct amount must be checked against the payment system.

A version that passes the release checks starts with a small production rollout, also called a canary. It receives only a portion of real tasks. The team continues to watch completion, payment outcomes that remain unknown, blocked permission violations, human takeovers, cost, and elapsed time. Once a person confirms a new incident, the same task and environment join the regression set: tests that check whether old problems have returned. The next version must pass them again.

Release Gate: The capability, reliability, security, cost, and latency requirements a candidate agent version must pass before a canary or production release.

Progress in 2026: Evaluation Covers the Entire Agent Run

In Anthropic's agent evaluation methodology published on January 9, 2026, the object being evaluated is the complete agent—the model together with its harness—rather than its final sentence. This changes how we write and grade tests. Reset the order and payment records before each trial, then let the agent execute from the beginning. Code checks payment records for the amount and duplicate refunds. Human judgment helps calibrate a model grader that assesses the explanation to the customer.

I also prepare two sets of questions. One contains difficult tasks the system cannot yet handle well, so we can see how much further its capabilities can go. The other contains failures we have already fixed and rerun after every change, so old problems stay fixed.

Google Agent Observability became generally available on June 18, 2026, connecting model calls, tool calls, errors, and elapsed time through OpenTelemetry records. The official July 29 update then brought online evaluation, Python or LLM graders, and observability into the same production feedback loop. When a refund slows down, its trace can show whether the delay came from the model, approval, or a payment timeout.

AWS's September 22, 2026 guide to Strands Evals and AgentCore turns the procedures taught by skill files into test cases. If the refund skill requires an order lookup first, the test checks whether that step appears in the actual execution record. OpenAI's Agent evals evaluate agents in batches using datasets, while trace grading checks the tools selected, arguments passed, handoffs made, and policies followed during a run. These tools provide batch testing and process grading. Our payment verifier queries the actual refund records to give the task its business-result grade.

Return to the Successful Demo

I now watch the demo with its test report beside it. A successful refund is one question in that report. Waiting for approval, a payment timeout, access to another company's order, and a failed notification must also run repeatedly and leave complete records.

If a version issues a duplicate refund again, the release checks block it. Only when results, paths, safety, and budget pass together does it enter a limited production rollout. “It seems to have become less capable lately” becomes evidence we can reproduce, compare, and use to fix the problem.

Once one agent has an established baseline, we can ask the next question: will splitting procurement research among several agents make it faster, or just introduce new ways to fail?