Failure Recovery: Checkpoint, Retry, and Rollback for Multi-Step Agents
When a step mid-chain fails, the steps before it may have already charged money, reserved a resource, or notified someone. Failure recovery is the discipline of deciding when to retry and when to roll back, so a partial failure doesn't turn into an expensive orphaned side effect.
Part of: Agentic Workflow Design: The 4 Decisions That Keep an AI Agent Safe in Production
A single-turn chatbot that gives a bad answer has an easy fix: the user clicks regenerate. A multi-step agent that fails halfway through doesn't get that luxury - by the time step three breaks, steps one and two may have already charged a card, reserved a resource, or notified someone downstream. The failure isn't just "the task didn't finish." It's "the task partly happened, and now something is in an inconsistent state."
When a Step Fails: Retry or Rollback
The Recovery Triad: Checkpoint, Retry, Rollback
Three primitives do the actual work of failure recovery. A checkpoint is a saved snapshot of state and progress at a specific milestone - if something breaks later, the system resumes from the last checkpoint instead of starting over from zero. A retry re-attempts the failed step itself, but only makes sense when that step is idempotent - safe to run twice without a duplicate side effect - and should space attempts out with exponential backoff instead of hammering an already-struggling service. A rollback, also called a compensating action, is a deliberate sequence that undoes or cancels the effects of steps that already succeeded, once a later step fails in a way that can't be recovered forward.
Running example: WarehouseRestockBot - an agent that automates store restocking: placing a purchase order with a supplier, reserving a loading dock slot for delivery, and notifying the destination store of the incoming shipment.
Not Every Failure Wants the Same Response
The mistake that breaks this the fastest is treating every failure the same way - "just retry three times" - regardless of why it failed. A request that timed out because of a flaky network connection is a completely different situation from a request that failed because the warehouse's loading dock has zero available slots for the requested week. The first is transient and will very likely succeed on the next attempt. The second is a deterministic rejection that will fail identically no matter how many times it's retried - the dock isn't going to have empty slots on attempt four that it didn't have on attempt one.
That distinction is what decides whether a failure should retry or roll back: is the step idempotent and is there retry budget left, or is this a business-logic rejection that no amount of retrying will fix? Get this wrong in the naive direction and the system burns compute retrying an outcome that was never going to change, while the parts of the workflow that already succeeded - a purchase order already placed with a supplier - sit unresolved the whole time.
Routing WarehouseRestockBot's Failures Through the Decision Tree
Same three-step chain - purchase order, dock reservation, store notification - three different ways it can break.
The supplier's order API times out from a network blip while placing the purchase order. An idempotency key was already generated for this request.
Safe to retry. The idempotency key guarantees a second attempt can't create a duplicate purchase order. Wait, then retry with exponential backoff - 1s, then 2s, then 4s.
Why Halting Isn't a Safe Default
The easiest failure-handling code to write is also the most dangerous one to ship: catch the error, log it, stop. It looks safe because nothing crashes further. It isn't safe, because stopping doesn't undo what already happened - it just freezes the workflow in a half-finished state and hopes someone notices. If WarehouseRestockBot places a purchase order, then fails to reserve a delivery dock, and the system simply halts there, the result isn't "no restock happened." It's "a supplier now has a live purchase order for inventory that has nowhere to be received" - an orphaned side effect that costs real money and real supplier relationship capital to unwind manually, and that nobody will notice until the truck shows up with nowhere to unload.
A Naive Halt Gets More Expensive the Later It Happens
Position of the failing step in the chain × side effects already committed = cost of just stopping
No prior step succeeded yet. Halting here costs nothing beyond the one failed attempt - there's nothing to unwind.
The purchase order is already live with the supplier. Halting here leaves ordered inventory with no dock scheduled to receive it.
Both the purchase order and the dock reservation are already live. Halting here leaves a delivery scheduled to arrive at a store that has no idea it's coming.
Step one is the easiest to test, so it's often the one that gets a rollback plan first. But it's the last step in the chain sitting behind the most already-committed side effects -rollback coverage matters most exactly where it's built last
These tiers describe how many upstream commitments are left dangling by a naive halt, not a measured dollar cost - the actual cost of an orphaned purchase order or an unused dock reservation depends on the specific supplier terms and warehouse operations, and should be estimated for your own workflow rather than assumed from this ordering.
Recovery Design Happens Before the Failure, Not During It
None of this - checkpointing, deciding which steps are idempotent, designing what a rollback actually undoes - can be improvised in the moment a failure occurs. It has to be decided at design time, step by step, before the workflow ever runs: which steps get a checkpoint, which steps are safe to retry and under what backoff schedule, and exactly what compensating action reverses each step if something downstream breaks. A workflow that reaches this decision for the first time while it's actively failing in production will improvise something worse than either a clean retry or a clean rollback - usually a partial, inconsistent one.
This is also where failure recovery leans on tool scoping more than it looks like it should: a rollback is only cheap to execute if the agent already has a scoped, well-defined tool for reversing that specific action - a cancellation endpoint, a compensating API - rather than needing broad, ad hoc access improvised after the fact to clean up a mess nobody designed for.
Common Pitfalls in Failure Recovery
A handful of mistakes recur once teams build their first multi-step agentic workflow: retrying a non-idempotent action and creating a duplicate side effect (a second charge, a second order) instead of a fix; retrying a deterministic business-logic rejection as if it were a transient error, burning time and compute on an outcome that was never going to change; treating "the retry budget ran out" as a stopping point instead of a trigger to roll back and escalate; and designing recovery only for the failure that happened in testing, leaving every other step in the chain with no rollback plan at all.
Get failure recovery right and a workflow can fail loudly and safely instead of quietly and expensively - every partial failure either resolves itself through a correctly-scoped retry, or unwinds cleanly through a rollback, instead of leaving an orphaned side effect for someone to discover days later.