Recover from Failures Without Duplicating Work

A missing reply does not tell you whether the requested action happened.

A coordinator asks a system to reserve one practice notebook. The provider saves the reservation, but the reply is lost. The coordinator's screen shows a timeout.

Should the system try again?

It depends on what “again” means. Creating another reservation could duplicate the work. Repeating the same identified operation through a provider that enforces duplicate prevention may be safe. Looking up the original operation may establish that nothing needs to be repeated.

Reliable recovery begins by separating a failed conversation from a failed effect.

Classify the failure before choosing a response

Some failures are temporary: a service is busy or a connection drops. Others require a change: an input is invalid, a credential lacks permission, or an identifier refers to the wrong account. Repeating the same invalid request does not repair it.

A third category is especially important: unknown completion. The request may have reached the destination, and the effect may have occurred, even though the caller did not receive confirmation.

Use a small decision table:

The service's actual semantics matter. An error code alone may not establish whether an operation began. Read the target API's documentation and retain the identifiers that let you investigate.

Give one intended operation a stable identity

An idempotency key identifies a logical operation across attempts. The destination uses that identity to avoid repeating the effect and to return a consistent result when appropriate.

A fresh random key for every retry defeats that purpose. Reusing one key for two different intended orders is also wrong. The identifier must be stable for the same intention and distinct for a new intention.

The payload matters too. If OP-001 originally meant one notebook, a later request for two notebooks under OP-001 should not silently become either a duplicate or an update. AWS's discussion of idempotent APIs explains the connection between caller-provided identifiers, retries, and changed intent.

Check how long the destination retains keys and which operations its guarantee covers. Duplicate prevention that expires after a retention window is not a permanent promise.

Keep a checkpoint before the uncertain step

A useful checkpoint records the operation identifier, exact payload or a protected reference to it, current state, and any provider receipt. It should survive the process that is doing the work.

For our fictional reservation, the initial state is pending. If the provider commits but the reply is lost, the state remains unresolved. On resumption, the workflow looks up the original operation and compares its content before marking the work verified.

A checkpoint is not proof that an external action happened. It records what the workflow knows. Keep unknown or pending available as truthful states rather than forcing every interruption into success or failure.

Run a lost-reply exercise

Save the following as recovery_lab.py and run it with Python 3.9 or later. Two temporary SQLite files stand in for the workflow journal and the provider's records. The provider is simulated locally; no order or payment is sent anywhere.

from contextlib import contextmanager
import json
from pathlib import Path
import sqlite3
import tempfile


@contextmanager
def database(path):
    connection = sqlite3.connect(path)
    try:
        yield connection
        connection.commit()
    except Exception:
        connection.rollback()
        raise
    finally:
        connection.close()


def provider_lookup(path, operation_id):
    with database(path) as connection:
        return connection.execute("SELECT payload, receipt FROM orders WHERE id=?",
                                  (operation_id,)).fetchone()


def provider_submit(path, operation_id, payload, lose_reply=False):
    with database(path) as connection:
        connection.execute("BEGIN IMMEDIATE")
        old = connection.execute("SELECT payload, receipt FROM orders WHERE id=?",
                                 (operation_id,)).fetchone()
        if old and old[0] != payload:
            raise ValueError("Identifier reused with different content")
        receipt = old[1] if old else "receipt-" + operation_id
        if not old:
            connection.execute("INSERT INTO orders VALUES (?, ?, ?)",
                               (operation_id, payload, receipt))
    if lose_reply:
        raise TimeoutError("Simulated reply loss after the provider committed")
    return receipt


def run(journal, provider, operation_id, order, lose_reply=False):
    payload = json.dumps(order, sort_keys=True, separators=(",", ":"))
    with database(journal) as connection:
        row = connection.execute("SELECT payload, status FROM jobs WHERE id=?",
                                 (operation_id,)).fetchone()
        if row and row[0] != payload:
            raise ValueError("Job identifier reused with different content")
        if row and row[1] == "verified":
            return "already_verified"
        if not row:
            connection.execute("INSERT INTO jobs VALUES (?, ?, 'pending', NULL)",
                               (operation_id, payload))
    found = provider_lookup(provider, operation_id)
    if found:
        if found[0] != payload:
            raise ValueError("Provider record has different content")
        receipt = found[1]
    else:
        try:
            receipt = provider_submit(provider, operation_id, payload, lose_reply)
        except TimeoutError:
            return "unknown"
    with database(journal) as connection:
        connection.execute("UPDATE jobs SET status='verified', receipt=? WHERE id=?",
                           (receipt, operation_id))
    return "verified"


def main():
    with tempfile.TemporaryDirectory(prefix="recovery-lab-") as folder:
        journal, provider = Path(folder) / "journal.db", Path(folder) / "provider.db"
        with database(journal) as connection:
            connection.execute("CREATE TABLE jobs (id TEXT PRIMARY KEY, payload TEXT, status TEXT, receipt TEXT)")
        with database(provider) as connection:
            connection.execute("CREATE TABLE orders (id TEXT PRIMARY KEY, payload TEXT, receipt TEXT)")
        order = {"item": "practice-notebook", "quantity": 1}
        assert run(journal, provider, "OP-001", order, lose_reply=True) == "unknown"
        with database(journal) as connection:
            assert connection.execute("SELECT status FROM jobs").fetchone()[0] == "pending"
        assert provider_lookup(provider, "OP-001") is not None
        assert run(journal, provider, "OP-001", order) == "verified"
        assert run(journal, provider, "OP-001", order) == "already_verified"
        try:
            run(journal, provider, "OP-001", {**order, "quantity": 2})
        except ValueError:
            pass
        else:
            raise AssertionError("Changed content was accepted")
        with database(provider) as connection:
            assert connection.execute("SELECT count(*) FROM orders").fetchone()[0] == 1
        print("PASS: lost reply reconciled; one provider record; changed content rejected")


if __name__ == "__main__":
    main()

The exercise ran successfully during preparation. The first attempt left one provider record and a pending journal entry. The next run found the existing provider record and marked the journal verified. A further replay returned already_verified. Reusing the identifier with a changed quantity was rejected.

The test demonstrates a lost acknowledgment after a committed effect. It does not simulate a real network, eventual consistency, multiple workflow workers, or a provider that lacks lookup and duplicate-prevention support.

Notice where the guarantee lives. The provider table has a unique operation identifier, checks payload equality, and records the effect within its own transaction. The workflow's local journal alone would not prevent duplicate effects in an unrelated external system.

Bound retries in more than one dimension

A retry policy should specify eligible errors, maximum attempts, delays, and an overall deadline. A timeout on each request can still permit an unreasonably long workflow if attempts continue indefinitely.

Backoff increases the delay between attempts. Jitter varies that delay so many clients do not retry in lockstep. These techniques can reduce pressure during temporary failures, but they do not make a non-idempotent write safe.

Avoid stacking independent retry loops without understanding the result. If three layers each make three attempts, the deepest service can receive far more calls than the outer application's “three attempts” suggests. Put retry ownership where it can account for the whole operation.

The demonstration makes at most one submit attempt per run and reconciles before another submit. It has no automatic network retry loop. That narrow behavior keeps the important recovery decision visible.

Handle an inconclusive lookup honestly

A real provider's “not found” response may be temporarily stale. If its documentation does not establish that the lookup is authoritative at that moment, absence may not prove the first write failed.

Use the provider's supported idempotency mechanism, operation status endpoint, or reconciliation procedure. If none provides enough evidence, stop in a review state. Do not turn uncertainty into another irreversible action merely to make the workflow appear complete.

Keep enough information for a person to investigate: operation identifier, intended effect, time of attempt, known responses, and relevant account or resource reference. Exclude credentials and unnecessary private content from routine logs.

Make cancellation and manual recovery explicit

Cancellation should stop new work, but it may not interrupt an external action already accepted by a provider. Record whether the current step was never started, stopped locally, or left with an unknown outcome.

A failed-item or review queue is a place for work needing attention, not a bin that makes failures disappear. Each item needs a reason, an owner, and a permitted recovery action. A person should be able to resume the original operation without inventing a new identity accidentally.

For a multi-step process, preserve completed steps and their outputs. If document creation succeeded but notification failed, recovery should not create a second document merely because the overall workflow is incomplete.

A reusable prompt

Design recovery for this workflow. Classify invalid input, permission failure, temporary failure, and unknown completion separately. Define stable operation identity, payload matching, checkpoints, retry ownership, deadlines, cancellation, and authoritative reconciliation. Inject a harmless lost reply and show that recovery preserves the existing effect without creating another one.

For students: learn from an interrupted experiment

Students can run the local demonstration without accounts or paid services. Explain why the timeout does not imply that the provider table is empty, and why the second run checks the provider before submitting.

The same reasoning applies to submitting a form, saving a research result, or running a long computation. Use a stable run identifier and preserve completed outputs. Follow the actual submission system's instructions instead of repeatedly clicking because a page is slow.

Practice: recover the original operation

Run the program and inspect the order of its assertions. Describe the journal and provider state immediately after the injected timeout. Then modify the fictional order under the same identifier and explain why rejection is the appropriate response.

Completion check: Recovery preserves the confirmed provider effect, uses the original identity, and leaves uncertainty visible until evidence resolves it.

Stretch: Add a cancellation flag between two harmless steps. Verify that the completed first step remains recorded and the second step does not begin after cancellation.