Building an Enterprise Agent Harness from Scratch2/10: Why Does an Agent Declare Victory Too Soon?

The agent has already written “Complete” in the conversation window. In the payment system, however, the transaction list has no refund ID. On the far right of the email list, the status still says “Draft.”

The sentence reached the finish line before the real world did.

The minimal loop in the previous chapter treated the model’s final response as the end of the program. We now need to separate two finish lines. The model may finish speaking. The task may finish only after the evidence is complete.

Turn “Take Care of It” into a Finish Line We Can Verify

The user originally said only, “Please take care of this duplicate charge.” That gives us a direction, but not a definition of done.

I break the request into parts. First, identify the customer, order, and two transactions that form the duplicate charge. Then define the maximum refund, actions that are out of scope, and conditions that require a person. Finally, specify the payment state that counts as a successful refund and the email state that counts as delivery.

After checking the transactions, we learn that the order was overcharged by $42. I put the open questions and this amount into a task contract. The minimum version contains at least the following fields:

If the goal changes during execution, the system creates a new contract version. The original amount, scope, and approval are not silently overwritten. The agent is no longer interpreting an open-ended sentence. It is working on a track with boundaries and a finish line.

Task contract: A structured definition of the goal, boundaries, capabilities, deliverables, and acceptance criteria fixed before execution.

Invariant: A condition that must remain true throughout execution, such as “the total refund cannot exceed the actual duplicate charge.”

Once the Finish Line Is Clear, Track How Far We Have Gone

The refund task has three work items: create one refund, write the refund ID into the CRM, and deliver a notification email to the order owner.

After each step, the runtime updates a separate ledger and attaches an evidence reference. “Separate” means that chat text does not control the status. If the payment API returns only “request accepted,” the refund item remains RUNNING. It becomes PASSED only after the payment system shows a matching refund ID in the final state required by the task contract. If the email is still in the drafts folder, a finished body does not count as delivery.

This progress ledger does not live inside the chat summary. It retains each work item’s status, attempt count, errors, and evidence after context compression, a model change, or a service restart.

The model proposes the next step. The runtime holds the pen that changes state.

In its experiment on effective harnesses for long-running agents, Anthropic uses a structured feature list, completes one item at a time, and marks an item complete only after the implementation passes a real test. A refund does not need to copy the software-development format, but it can use the same principle: define the finish line first, save progress independently, and require evidence before passing an item.

Progress ledger: A runtime record, independent of chat text, that stores work items, states, attempt counts, errors, and evidence references.

The Final Signer Cannot Rely on the Worker’s Own Report

The easiest “evidence” for the worker agent to obtain is the content it just created. It rereads the refund email, sees the correct amount and tone, and declares the notification complete. That proves only that a draft exists. It does not prove that the email system delivered it.

We need an independent verification program: the Verifier. It follows the task contract to three sources of truth. It queries the payment system for exactly one $42 refund on this order and confirms that the record has reached the state required by the contract. It checks the customer ticket system (CRM) for the same refund ID. It asks the email system for the recipient and delivery state. Deterministic rules check amounts, states, and counts. A model score or human review can then evaluate tone and explanation quality.

The Verifier has read-only access. It can inspect a refund record, but it cannot issue another refund. It reads authoritative systems instead of asking the worker agent to confirm its own summary.

Three gates check the plan, process, and outcome. The plan gate asks, “Does every required deliverable have evidence?” The process gate asks, “Did tools, approvals, or budgets cross a boundary?” The outcome gate reads the payment, CRM, and email systems and asks, “Did the real world change?” Only after all three gates pass does the task enter COMPLETED.

Verifier: A component or role that independently checks whether a task is truly complete against predefined criteria and evidence from the environment.

Ground truth: A verifiable fact from the target system, test result, or human confirmation.

After the Model Says “Done,” the State Machine Finishes the Work

When the model submits its final answer, the Harness does not close the task. It moves the state from running to verifying: RUNNING → VERIFYING.

If user input is missing, the task pauses and asks a question. If a high-risk refund lacks approval, it waits for approval. If the payment system is still processing, it waits for evidence. If the external outcome cannot be confirmed, it enters reconciliation. If the budget is exhausted, it stops at STOPPED_WITH_EVIDENCE and hands the existing progress and evidence to a person.

A Boolean value gives us only two answers: complete or incomplete. It cannot tell us where the task is stuck. A state machine defines the allowed states and where each state may go next. Every “not complete” now has a reason: waiting for the user, approval, the payment result, or reconciliation. The task will not generate another refund while waiting for payment, and it will not hide a missing approval behind a fluent answer.

2026 Progress: Frameworks Separate “This Turn Ended” from “The Task Succeeded”

In its agent evaluation guide published on January 9, 2026, Anthropic separates the execution trajectory from the final outcome. “The refund is complete” and the tools selected by the agent belong to the trajectory. The refund record in the payment system is the external outcome. This maps directly to the process and outcome gates above.

In the OpenAI Agents API quickstart, turn.completed means only that the current turn ended. session.idle means only that no work is running now. Neither can declare success for the business system. OpenAI’s trace grading can evaluate tool selection, arguments, handoffs to another agent, and policy execution. The Verifier must still query the payment system for the refund ID.

When a refund needs human approval, LangGraph can save the current node and use interrupt() to wait. A node is one step in the workflow. The approver can close the browser and return the next day; the application still loads the same order, amount, and action. On resume, the program re-enters the paused node to process the approval result. The real refund call therefore belongs in a separate node after it and always uses the same idempotency key. OpenAI Agents SDK provides approve and reject controls. Restate can persist the waiting state. These tools reduce the code required for pausing, resuming, and controlling the loop. The team still defines the task contract and business acceptance criteria.

An approval must remain bound to the original action. If the approver changes the amount, the system rejects the old call, creates a new action ID, and runs policy and approval again. It never uses an old approval to release a refund that has changed.

Return to the Opening Claim: “Complete”

The agent submits its final answer again. This time, the runtime first opens the task contract. The payment item has no refund ID, so it keeps waiting. The CRM item has no record, so it returns to execution. The email is still a draft, so it cannot pass the outcome gate.

Only after the payment, CRM, and email systems each provide matching evidence does the state move from VERIFYING to COMPLETED. “Complete” is no longer a sentence generated by a model. It is a set of facts that the program can check again.

The finish line is finally stable. The task will now run longer, and its history, tool results, and files will keep growing. The next chapter addresses a new problem: the most important rule is still in the context, so why does the agent forget it?