Failure, Recovery and Retry
What Happens When an AI System Does Not Get the Result It Expected?
The Happy Path Is Only One Path
Our earlier agent loops often looked clean:
Real systems are messier. A tool may time out. Permission may be denied. A file may have changed. A database may reject a query. A model may choose the wrong tool. A service may return malformed data. An operation may succeed while its response is lost.
Failure is not one event. It is a class of outcomes occurring at different layers of the system.
First Question: What Failed?
Decision Failure
The model chooses an inappropriate action, arguments or interpretation.
Authorization Failure
The requested operation is denied by policy, credentials or human approval.
Invocation Failure
The tool cannot start, rejects arguments or reports an internal error.
Communication Failure
A request or response is delayed, lost, interrupted or malformed.
External-System Failure
The database, service, host, sensor or other target does not behave as expected.
Outcome Failure
The operation completed, but verification shows the intended result did not occur.
Expected Result Versus Observed Result
An agentic system often operates with an implicit or explicit expectation.
If they differ, the system has learned something important. But it should not immediately conclude why.
Tool result: command returned success
Expected: service running
Observed: service still failed
Conclusion: intended outcome not achieved; cause requires further evidence.
Successful execution is not the same thing as successful outcome.
The R/W/X → ΔState Chain
Our permissions work gives us a useful failure-analysis chain:
A failure can occur at every arrow.
| Stage | Question | Example |
|---|---|---|
| Request | Was the intended action represented correctly? | Wrong hostname supplied. |
| Authorization | Was the action permitted? | Permission denied. |
| Execution | Did the operation actually run? | Tool process crashed. |
| State Change | What changed? | File written, service unchanged. |
| Verification | Did the desired outcome occur? | HTTP still unavailable. |
Retry Is Not “Try Again Until It Works”
A retry is another attempt at an operation after a previous attempt did not produce a usable confirmed result.
The middle question is critical.
Safe Retry: The Easy Case
Suppose a read-only status request times out before returning data. Repeating it may be reasonable because the operation is observational and inexpensive.
Attempt 2: service_status(web01) → httpd failed
The retry produced a usable observation.
Ambiguous Failure: The Dangerous Case
Now suppose an agent requests creation of a ticket:
The agent sees a timeout. Did creation fail—or did creation succeed and only the reply fail?
Blindly retrying might create a second ticket.
“I did not receive confirmation” does not mean “the operation did not happen.”
Idempotency
An operation is idempotent when repeating the same intended operation does not create additional unintended effects.
A system can use an operation identifier:
Attempt 1: response lost
Attempt 2: same operation_id
Server: recognizes operation already completed; returns original result instead of creating another ticket.
Idempotency makes retries much safer, especially across unreliable networks.
Retry Identity and Lineage
WITNESS should not record three attempts as three unrelated actions.
├── ATTEMPT A1 — timeout
├── ATTEMPT A2 — timeout
└── ATTEMPT A3 — success
The attempts share one task identity while retaining separate timestamps, inputs, outputs and errors.
When Not to Retry
Permission Denied
Repeating the same unauthorized action usually changes nothing.
Bad Arguments
Retrying malformed input without correcting it repeats the same defect.
Permanent Failure
A nonexistent resource may require a different plan, not more attempts.
Unsafe Ambiguity
If an earlier state-changing operation may already have succeeded, verify before repeating.
Retry With Change
Sometimes the next attempt should deliberately differ from the first.
That is not merely retry—it is recovery reasoning.
Observation: service outage
Recovery: use approved secondary source
Attempt 2: retrieve secondary source → success
Report: note fallback provenance.
Backoff — Do Not Hammer a Failing System
If a service is temporarily overloaded, retrying immediately and repeatedly can make the problem worse.
This simplified pattern is called exponential backoff. Real systems may add limits and randomized delay so many clients do not retry in lockstep.
Timeouts Are Decisions
A timeout does not prove that the remote operation failed. It says the waiting component stopped waiting after its configured interval.
Timeout is an observation about waiting—not necessarily an observation about remote execution.
This distinction is extremely important for forensic reconstruction.
Recovery Can Mean Rollback
Suppose a multi-step operation changes several resources and fails halfway through.
Possible responses include:
Continue
If remaining work can safely proceed another way.
Retry C
If the failed step is safely retryable.
Rollback
Attempt to restore earlier state.
Stop and Escalate
Preserve evidence and request human intervention.
Rollback itself is another state-changing operation and can fail. “Undo” is not magic.
Partial Success
Many real outcomes are neither success nor failure.
Read successfully: 94
Permission denied: 4
Missing: 2
A truthful system should preserve the partial result rather than compressing it to either “worked” or “failed.”
Recovery in a Multi-Agent System
Suppose Agents A, B and C are investigating in parallel and C fails.
The orchestrator might retry C, assign the task to another agent, use A/B if sufficient, change tools, wait for recovery, or escalate to a human.
Failure therefore changes the dependency graph.
Reassignment Is Not Independent Evidence
If C fails and the same task is handed to C2 using the same model, same source and same assumptions, the second result may improve availability—but it is not automatically independent verification.
Recovery and verification are different goals.
Checkpointing Long Jobs
A long-running agentic mission may preserve progress so a failure does not require starting from zero.
RECOVERY → LOAD CHECKPOINT → RESUME
This checkpoint is task/system state—not necessarily a model-training checkpoint. Same word, different layer.
Failure Can Corrupt Context
Imagine a tool returns malformed or misleading data and the agent accepts it as valid. That result can enter context and influence later reasoning.
Therefore validation should happen as close as practical to the boundary where data enters the system.
The Model Can Be Wrong About Failure
A model may interpret an unfamiliar tool response incorrectly, assume a timeout means failure, or confidently propose a recovery that is inappropriate.
Recovery policy should not depend solely on fluent model interpretation when machine-readable status, explicit error codes, transactions or external verification are available.
Stop Conditions Matter
Without limits, an agent can enter an expensive or useless loop.
Useful controls include maximum attempts, maximum elapsed time, cost/token limits, repeated-error detection, escalation rules and human approval.
Knowing when to stop is part of recovery design.
WITNESS: Failure Is Evidence
A failed attempt should not disappear merely because a later retry succeeded.
The failure may explain later behavior. Removing it from the record destroys causal history.
Known Unknown: Did State Change?
One of the most important forensic outcomes may be:
If a state-changing request was sent and the connection died before confirmation, WITNESS should preserve that uncertainty until independent evidence resolves it.
Do not silently turn an unknown into either success or failure.
A Recovery Decision Tree
↓
WHAT FAILED?
↓
WAS EXTERNAL STATE POSSIBLY CHANGED?
↓
VERIFY IF NEEDED
↓
IS RETRY SAFE AND USEFUL?
↓
RETRY / REVISE / FALL BACK / ROLLBACK / ESCALATE / STOP
↓
VERIFY FINAL STATE
The Boundary
A failure may occur in the model decision, authorization, tool, transport, external system or outcome.
A timeout does not prove the remote operation failed.
A successful command does not prove the intended outcome occurred.
Retry is an engineering decision, not a reflex.
State-changing operations require special care when previous success is uncertain.
Recovery can include retry, revised action, fallback, rollback, reassignment, escalation or stopping.
Failures and retries belong in the evidence record.
When the result is unexpected, first determine what is known before deciding what to do again.
HAAW Checkpoint
Failure: An operation or system outcome that does not provide the required usable result.
Retry: Another attempt at an operation after an earlier attempt did not yield a usable confirmed result.
Recovery: The broader process of responding to failure and restoring progress or a safe known state.
Timeout: A local decision to stop waiting after a configured interval; not proof of remote failure.
Idempotency: A property allowing an intended operation to be repeated without duplicating its effect.
Backoff: Deliberately increasing delay between retries to reduce load and allow recovery.
Rollback: An attempt to reverse earlier state changes after partial failure.
Partial Success: An outcome in which some requested work succeeded and some did not.
Checkpoint: Preserved task/system progress from which work can potentially resume.
Attempt ID: Identity distinguishing individual attempts while linking them to the same task.
Unknown State: A condition where available evidence cannot yet establish whether an external change occurred.
Central lesson: Retry only after asking what failed, what may already have changed, and whether repeating the operation is safe.
Planning and Replanning — How an AI System Turns a Goal Into Steps and Changes Course When the World Disagrees