Know What Happened—and Recover When the Workflow Fails

The screen says the request timed out. Did nothing happen, or did the action complete before the response disappeared?

Those two situations look similar to the caller. Repeating the action can be harmless in one case and create a duplicate in the other.

A reliable workflow keeps enough evidence to distinguish them. Your work product is an operating scorecard, a failure table, and a recovery exercise that demonstrates what happens before and after an action commits.

The supplied tests execute a local SQLite simulation with fictional Mesa records. No model, remote provider, customer message, payment, or live business record is involved.

Track the business operation, not just the program run

A program can start successfully and still fail to finish the business job. A tool can return valid JSON while producing an unusable answer. A workflow can stage a draft correctly while the customer still needs help.

Define the completion event for each scope. In the practice workflow, completion is staged_local_review_only: the review queue, operation record, and action audit agree that one item was staged. It does not mean the answer was sent or the customer case resolved.

Use a stable operation identity across retries. Give each attempt its own timing and diagnostic information, but keep it linked to the same intended action. Otherwise, the dashboard may count three attempts as three completed jobs.

Preserve the actor, record version, source version, approved proposal, action, destination, result reference, and unresolved status needed to reconstruct the work. Keep secrets and unnecessary personal content out of routine logs.

Name the outcomes that require different responses

Do not hide these states behind one red or green indicator. The response depends on what the evidence establishes, not simply on whether a network call raised an error.

A rejected request may be the system working correctly. Count it separately from an outage or bad model answer. An attempted unauthorized send should not make the workflow look unreliable merely because it refused to send.

Exercise failure before the commit

The local test starts with an exact simulated approval. It inserts the review item, then deliberately raises an exception before the operation receipt and action audit can commit.

Because those action writes belong to the same transaction, the test verifies zero queue rows, zero operation rows, and zero action-audit rows afterward. The simulated approval itself remains a separate earlier record. A later valid attempt can stage the item once.

This demonstrates rollback within the defined local database boundary. It does not show that an email, payment, or other external action can be rolled back by reversing local database work.

When examining a real failure, identify each side effect and its boundary. A local transaction can protect the records it controls. It cannot make an unrelated provider's operation disappear.

Exercise a lost acknowledgment after the commit

The second test commits the review item, receipt, and audit row, then simulates losing the response. The caller sees an error even though the local action exists.

The test reopens the database and submits the same operation again. The workflow returns the original receipt with a replay indicator. The counts remain one queue row, one operation row, and one action-audit row.

It performs this lookup after the simulated approval has expired. That is intentional: it is reporting an action already completed, not using expired authority to create another one. Current access to the result still needs to be enforced in a real service.

AWS's idempotent API guidance explains why an explicit request identity and consistent handling of repeated intent matter. The practice database keeps its operation records for the exercise's lifetime. A production design also needs a retention policy long enough for its retry, reconciliation, and business requirements.

Reconcile uncertain external outcomes

Suppose a future connector sends a message and the response times out. First inspect the provider's supported operation lookup, message record, or idempotency mechanism. Determine what evidence can establish whether the original action completed.

Do not infer failure solely from the missing acknowledgment. Do not mark success because the model says the message probably went through. Preserve an unknown outcome when neither conclusion is supported.

If a provider has no reliable lookup or duplicate-prevention mechanism, constrain the workflow accordingly. Some uncertain actions may require manual reconciliation. Blind automatic retries are not a substitute for evidence.

Record partial results in a multi-step workflow. If creating a record succeeded but attaching a document failed, retry or repair the attachment step as appropriate; do not recreate the entire business object without checking what already exists.

A compensating action is also a real action. Sending a correction, issuing a refund, or reversing an entry requires its own valid scope and authority. Recovery should not quietly grant broader powers than the original workflow had.

Retry only the failures a retry can help

A temporary service interruption may improve after a delay. A missing permission, invalid field, revoked approval, or changed record will not usually be fixed by submitting the same request more aggressively.

AWS's guidance on controlling and limiting retries recommends bounded retries and backoff, with jitter to reduce synchronized retry pressure. Apply provider-specific instructions and distinguish retryable failures from conditions that require a decision or correction.

Set a maximum number of attempts, a total time or resource budget, and a destination for unresolved work. Account for retries inside SDKs and connectors so several layers do not multiply the number of attempts unexpectedly.

The local example does not implement an automatic retry scheduler or wait loop. The test harness deliberately invokes a second attempt to check the duplicate and recovery behavior. A production scheduler would need separate limits, timing, cancellation, and operational visibility.

Monitor quality and effort alongside availability

A workflow that runs all day can still give bad answers. Build a scorecard with the outcomes needed for the business decision.

Track eligible cases, accepted output, first-review acceptance, corrections, unresolved exceptions, and the complete effort spent. Include model and tool costs where they exist. Distinguish successful tool calls from useful results and completed customer outcomes.

Measure review load as volume changes. A draft workflow may work for ten requests and overwhelm the reviewer at a hundred. Queue age and unassigned exceptions can reveal that problem before an average response-time number does.

Do not turn twenty passing fixture cases into a claim of 100% production reliability. The practice results establish the behavior of this code for these inputs. They contain no model calls, live traffic, concurrent-worker experiment, or provider outage test.

The model evaluation plan is a separate file with blank observed results. It lists policy accuracy, missing availability, injected instructions, malformed output, source conflict, and context boundaries as cases to evaluate when a real model is introduced.

Review changes as changes to the workflow

A model update, prompt revision, source edit, new tool, permission change, or connector release can alter behavior. Keep a versioned record of the parts that produced the result.

Before expanding a change, rerun the cases most likely to reveal the affected risk. A source edit needs source and answer checks. A permission change needs access checks. A new sending adapter needs action, duplicate, and uncertain-outcome tests.

Avoid running a huge unrelated test suite just to produce a reassuring total. Select tests from the actual change and the remaining risk, then record their practical limits.

NIST's AI Risk Management Framework supports ongoing consideration of trustworthiness in AI design, use, and evaluation. It is voluntary guidance, not a certification that this workflow or any future integration is safe in all settings.

Keep the earlier result and the changed result available for comparison. Do not silently replace the baseline or edit the old expected answer until a regression disappears.

Make pause and fallback real operating capabilities

Define who can pause new actions, how the pause is enforced, and who owns the unresolved queue. If work is already in flight, determine which actions can still complete and how their results will be reconciled.

The practice workflow has an in-memory pause flag that blocks new staging. The tests exercise it in one process. It resets on restart and is not a durable production emergency control.

A production pause should survive the failures it is meant to contain and be respected by every relevant worker. Its exact design depends on the system. Test the actual implementation, including restart behavior and the state of work already accepted.

For Mesa's narrow task, the manual fallback remains the current policy and an appropriate reviewer. Preserve the unanswered availability question. Do not let fallback become permission to confirm an appointment without checking the calendar.

Write a short incident record

Capture what was requested, what is known to have happened, what remains uncertain, the affected scope, containment, responsible owner, and next check. Link the relevant records instead of copying unnecessary private content everywhere.

Prepare a draft incident note from the supplied operation records.
Separate attempted, rejected, committed, replayed, and uncertain actions.
Identify evidence for each status and preserve missing information.

Propose the next reconciliation or recovery step with its owner and
required authority. Do not invent a completed action, customer notice,
refund, reversal, approval, or successful restart.

After recovery, identify the cause and the change that would prevent or expose a similar failure. Verify that change with an appropriate case. Keep the incident and its resolution connected so the team can learn from the actual sequence.

An automation earns trust when people can tell what it did, what it did not do, and how to continue when it stops. That operational clarity matters as much as the first impressive answer.

Next: Part 10 covers governance and improvement: policy, vendors, meaningful impact, continuity, and a practical ninety-day growth plan.