Building an Enterprise Agent Harness from Scratch5/10: Why Can’t You Simply Retry a Failed Agent?

The procurement agent has just handed an approved request-for-quote email to the sending service. Ten seconds pass. The tool returns only two words: “Request timed out.” There is no message ID and no reason for failure.

The agent treats the timeout as a failure and sends again. The vendor opens its inbox and finds two identical emails.

The problem lives inside those ten seconds. The missing response tells us only that the response did not arrive. It does not tell us where the first email went. The first step in recovery is therefore not to try again. It is to recover the real-world outcome of the original action.

After a Timeout, Find the First Email

I give the email a stable request ID and an idempotency key. They have different jobs. The request ID lets us trace and query where the original send went. The idempotency key tells the sending service, “This is still the same business action. Do not create another email.” Both values are saved before the request leaves. If the service later returns an operation ID or message ID, we add it to the ledger. Even if the success response is lost, we can query the original request, and the service can recognize a duplicate call.

Read the diagram from top to bottom. After Timeout, the system first uses Query sending service. If the query finds a message ID, the left branch saves the receipt. If the request is still processing, the system waits and queries again. It retries with the original idempotency key only when the service confirms both that no effect occurred and that the old request has stopped. If the authoritative system still cannot determine the outcome, the task goes to human reconciliation.

We must separate several kinds of “recovery.” If a request never left the current program, it can retry within attempt and time limits. If a model output is missing a field, the system returns a specific format error for correction. If the program crashes after saving a decision, it reads the saved history on restart and continues with the same decision. If the task is waiting for approval, it returns to the same waiting point. If the request left the system but no result came back, the system must reconcile with the authoritative service. The most important boundary separates an ordinary transient failure from “the external action may already have happened.” The first can retry within limits. The second must query first.

Retry: Attempting the same step again after confirming that it is safe to do so, with limits on attempts, time, and backoff.

Replay: Reconstructing control flow from durable history and continuing with decisions already recorded.

Reconcile: Querying the authoritative system for the real outcome of an external action before deciding to confirm, wait, compensate, or retry.

After the Process Restarts, How Does It Recognize the Same Email?

We need to query the email service, but we also need to preserve the agent’s own progress. The send times out, and then the program running the agent restarts. The chat is still present, but it does not tell us whether the model decided to send, whether the manager approved, or whether the tool call left our system. Chat records what was said. Durable state must also record which step was committed for execution.

After the model makes a decision, I save the decision and tool intent before touching the external system. The minimum record captures four facts in order: the email content has been fixed; the manager approved this content; the call with the stable request ID and idempotency key was dispatched; and the email service eventually returned a service-confirmed message ID.

The stable action ID remains the same after a process restart or a move to another machine. The recipient, subject, and body are serialized with a fixed field order and character encoding, then used to calculate a content fingerprint. This “digest” is a stable fingerprint computed from content, not a model-written summary. If the recipient, subject, body, or another material field changes, the fingerprint no longer matches. Approval binds to this fingerprint. Repeated clicks, two browser tabs, or two recovery attempts cannot create a new action. One atomic database operation records that “this approval has been claimed by this action,” and the send always reuses the original request ID and idempotency key.

When the process wakes up, it reads the record in order. If the decision is already saved, it does not ask the model to generate the email again. If approval has arrived, it proceeds to sending. If the call has left the system, it queries with the original request ID. If a service-confirmed message ID is already present, it confirms completion directly.

If the tool argument format, policy, or agent version has changed, the old record enters migration, reconciliation, or human review. Recovery continues a timeline that the system has already committed to. It does not quietly rerun the task under different rules.

Checkpoint: A durable snapshot of execution state from which a task can resume.

Durable execution: The ability of a task to continue from saved state after a process crash, deployment, or long wait.

The Most Dangerous Gap Comes Immediately After a Successful Send

There is a harder case. The email service has sent the first email and is about to return the service-confirmed message ID. The process crashes before that result is written into the agent’s record.

The lower lane in the diagram is the email service. It sent the message and kept a send record. The agent ledger in the upper lane crashed before receiving the receipt, so it knows only that the request was dispatched. After restart, it does not send another email. It queries the lower lane with the same request ID, recovers the original message ID, and confirms the result.

After recovery, the timeline stops at “request dispatched.” The agent knows that the request left its boundary, but it does not know that an email already exists in the outside world. A checkpoint for the task as a whole is not enough. We must keep a separate ledger for every action that may change an external system. A “side effect” is such a real-world change: sending an email, creating a purchase request, or making a payment.

I call this ledger the EffectLedger. It saves the stable action ID, email content fingerprint, and idempotency key before the call. When the system is ready to send, it advances the ledger to “dispatched” before calling the service. After the call, it records confirmed, unknown, or compensation required. Even if the program crashes after writing “dispatched” but before the actual send, recovery queries the original request first and continues only after confirming that a send is safe. The action ID is derived from the immutable task, business operation, logical step, and target resource. No matter how many attempts run or which machine runs them, the ledger still refers to the same action.

When the ledger is “dispatched,” the next step is a query. If the service confirms success, save the message ID. If it is still processing, keep waiting. If the outcome differs from the intent, enter compensation. The ledger records “retry allowed” only after the authoritative system confirms that the old request cannot continue and proves that it produced no effect. A new send still uses the original action ID and idempotency key.

Put the Recovery Rules into One State Machine

The state moves from planned to authorized, then to dispatched. After the service confirms the send, it ends at confirmed. If the service provides no definite result, it enters outcome unknown. From there, it may move only to confirmed, safe retry, or compensation required.

Two worker machines may recover at the same time and both try to send the email. I let them update the same ledger row with one condition: “Change this row to dispatched only if it is still authorized.” The database allows only one update to succeed. The other worker sees that the state changed and stops. This operation—confirm that no one changed the old state, then update it—is called compare-and-set. It prevents two workers from claiming the same action. Every transition also stores its basis, such as an approval record or an email-service query receipt. A dispatched action cannot move back to planned, because that would erase the fact that the request may have left the system.

The idempotency key must reach the service that actually sends the email. The service must register the key and create the send job as one atomic operation. When the same key arrives again, it returns the result of the original job. If the target service supports neither idempotency keys nor queries by key, the EffectLedger can record what the agent attempted, but it cannot invent the missing external outcome.

EffectLedger: A side-effect ledger that records the intent, authorization, dispatch, outcome, and reconciliation state of an external call under a stable action ID.

Idempotency: The property that executing the same logical operation once or repeatedly produces the same final business effect.

“Outcome Unknown” Is an Open Case, Not an Error Message

“Not found” cannot immediately become “not sent.” A read replica may not have caught up. The old request may remain in a queue. The sending service may be temporarily unable to answer queries. An unknown outcome therefore continues through authoritative queries. If the resource appears, confirm it. If the service proves no effect and confirms that the old request stopped, schedule a safe retry. If an existing result differs from the intent, enter compensation. If the system still cannot determine the outcome, send the case to human reconciliation.

If the wrong email was sent, compensation may be a clarification email. If a purchase request was created, compensation may be a new cancellation action. If money was paid, compensation enters a refund process. Compensation does not delete the original history. It uses a new business action to bring the system back into an acceptable state.

Outcome unknown: A state in which the system cannot confirm whether an external operation produced a side effect.

Compensation: A new business action that offsets or corrects the effect of a completed step that cannot be rolled back directly.

2026 Progress: Which Tools Can Handle This Recovery Work?

At this point, you may be wondering whether all of this bookkeeping, waiting, and recovery must be written from scratch. In 2026, more agent frameworks began connecting directly to mature durable execution systems. Existing agent code can therefore avoid part of the integration work for cross-process recovery, approval waits, and long-running task state.

The first path is Temporal, a durable workflow engine. “Durable” means that task progress remains in external storage after the current program exits. Temporal treats a purchase request as a Workflow and external steps such as querying vendors, calling a model, and sending email as Activities. When a generated email needs manager approval, Temporal saves the Workflow’s progress and releases the current process. When the approval signal arrives, it continues from the waiting point. If the service restarts, it uses history to recover unfinished steps.

Temporal moved its OpenAI Agents SDK integration to general availability in March 2026. Its October 26, 2026 product update also announced generally available integrations for Google ADK, OpenAI Agents SDK, AWS Strands, LangGraph, and others. These integrations connect runtime history, timers, approval signals, and recovery entry points. The email-sending Activity still uses the original request ID, idempotency key, and EffectLedger. Temporal restores workflow progress. The email service confirms the external outcome.

The second path combines PydanticAI with Restate. PydanticAI is a Python framework for building agents with explicit types for model inputs, tool arguments, and outputs. For this email, the recipient, subject, body, and action ID form one verifiable send intent. Restate sits in the execution layer and writes every model call, tool step, and routing result into a durable record called a journal. When the task must wait for manager approval, an awakeable waiting point saves progress and continues after a restart.

The Pydantic and Restate integration announced on May 12, 2026 connects the two layers. The official example uses RestateAgent(agent) to wrap an existing PydanticAI agent, run_typed to execute typed steps, and an awakeable waiting point to preserve approval. PydanticAI defines what an agent step receives and returns. Restate records how far the step progressed and how to continue after a crash. In the opening failure, Restate first recovers the completed steps. The program then queries the original email request. The sending service deduplicates by the original idempotency key. The records on both sides meet at this point.

If the project already uses LangGraph or Microsoft Agent Framework, it can use checkpoints to save the current orchestration position and apply the same email-query and safe-retry rules.

Workflow: A complete business task that must continue across several steps, waits, or failures.

Activity: One Workflow step that calls a model, API, or other external service.

Human-in-the-loop (HITL): Pausing a task before a critical action so that a person can approve, reject, or add input before it continues.

Return to the Ten-Second Timeout

Look again at the original timeout. The first send uses a stable request ID and fixed idempotency key. Ten seconds pass without a response. The ledger leaves the action at outcome unknown and does not create a second action.

After restart, durable history restores the same email intent. The runtime queries the sending service with the original request ID. If it finds a service-confirmed message ID, it records confirmed, and the vendor does not receive another email merely because the agent recovered. If the request remains in the queue, it waits. If the service confirms that the old request cannot continue and proves that it produced no effect, the system sends again with the same request ID and idempotency key. If the result still cannot be determined, the task stops for human reconciliation.

This is what recovery actually solves: the program can continue without hiding uncertainty inside a blind retry. In the next chapter, we will ask whose authority the recovered agent represents, what it may see, and what it may change.