New: Boardroom MCP Engine!

Governance and growth · Chapter 39 of 40

Keep the Business Running When AI Fails

Plan minimum service, recovery objectives, manual fallback, backups, uncertain actions, reconciliation, and a practical continuity exercise.

Updated

Open this chapter’s practice pack

On this page

Imagine arriving at work to find that your assistant will not load. The customer inbox still works, but the review queue is inaccessible. One message may have been sent just before the failure. A colleague has started recording new requests in a spreadsheet. The provider's status page says it is investigating.

Your first job is to preserve a useful level of service, protect records, and establish what has already happened. Waiting for the assistant to return is only one part of recovery.

Your deliverable is a continuity runbook and a tabletop exercise. A tabletop is a discussion of a constructed scenario. It can expose missing decisions and inaccessible information, but it does not prove that a backup restores, a fallback account works, or a provider integration can recover.

Define the minimum service you need

List the business functions that depend on the workflow. Identify which must continue immediately, which can wait, and which should stop until accurate information is available. Put a business owner beside each decision.

For fictional Mesa, a proposed minimum service is to record incoming requests and consult a controlled copy of the current inspection-fee policy. That can support a limited response while live appointment availability is unavailable. It does not justify confirming a booking from office hours or an old calendar export.

The preserved policy remains USD 45, with a credit toward a repair approved within thirty days of inspection. Any fallback copy needs a version, an owner, and a way to determine whether it is still current. If staff cannot establish that, they should record the question and seek confirmation instead of improvising a price.

Consider the whole dependency chain: identity provider, network, email, CRM, source documents, integration service, model provider, review queue, and the person who approves exceptions. A backup AI product is of little help if both products depend on the same unavailable identity account or business database.

Choose recovery objectives for a named service

A recovery time objective, or RTO, specifies the acceptable recovery duration for a defined system capability before business impact becomes unacceptable. State the capability and when the clock starts. Recovering minimum intake service is different from restoring all automated actions. NIST glossary: Recovery Time Objective

A recovery point objective, or RPO, identifies the point in time to which data must be recovered after an outage. Businesses often express it as a maximum tolerable interval of data loss. It concerns recoverable data, not simply when a file was last copied. NIST glossary: Recovery Point Objective

For the teaching exercise, propose a sixty-minute RTO for minimum intake service and a fifteen-minute RPO for the request register. These are illustrative targets, not adopted Mesa commitments or recommendations for every business. A different workflow may require substantially different objectives.

Record the reason for each target. Consider customer commitments, transaction volume, available staff, and how much uncertain work you could reconstruct. If the affordable recovery method cannot meet the required target, change the workflow or its dependency before relying on it.

Prepare the materials that recovery actually needs

Keep a controlled recovery pack accessible through an approved route that does not depend entirely on the failed service. Include the current runbook, essential contacts, permitted manual forms, source-version information, system dependencies, and the procedure for obtaining authorized access.

Do not put passwords or unrestricted customer exports into an ordinary shared document for convenience. Recovery access should remain limited and auditable. Identify who can retrieve protected backups and who can authorize their use.

Back up the records needed to reconstruct business state, not just the prompts used to generate text. Depending on the workflow, that may include pending items, source references, stable request identifiers, approvals, operation receipts, and exception history. Document omissions such as attachments or links that expire outside the original system.

NIST SP 800-34 Rev. 1 describes information-system contingency planning, recovery procedures, and validation before normal operation resumes. It is federal system guidance published in 2010; this chapter adapts those general planning ideas to a small-business exercise rather than presenting it as a current product configuration guide or universal business requirement. NIST SP 800-34 Rev. 1

Use a runbook with a clear activation decision

The first page should answer who can activate the fallback, who can stop automation, and how staff find out. Include triggers such as loss of a required source, an unavailable review queue, uncertain external actions, or behavior that no longer meets the workflow's limits.

A practical response sequence is:

  1. Record the start time, affected capability, and known scope.
  2. Stop new automated actions through the actual operational control.
  3. Preserve available logs and identify actions with uncertain outcomes.
  4. Assign an incident lead and activate the permitted minimum service.
  5. Record new work with stable identifiers in the manual register.
  6. Reconcile state before authorizing normal operations to resume.

This is a proposed sequence for adaptation. The local Part 9 pause flag is an in-memory teaching control, so it cannot serve as a production-wide stop mechanism. A real deployment needs a durable control covering its actual workers, queues, and integrations.

Avoid repeatedly changing settings while multiple people troubleshoot. Assign one person to coordinate changes and record what was changed, by whom, and why. Otherwise recovery can create a second problem that is harder to distinguish from the original failure.

Walk through a constructed outage

The companion scenario uses September 14, 2026, with a fictional local clock at UTC−06:00. No outage or staff exercise actually occurred.

Scenario timeSupplied eventWhat remains to establish
08:45Candidate request export checkpointWhether it is complete and restorable
08:52A request is created after that checkpointWhether another authoritative record preserves it
08:58An external message has an uncertain outcomeWhether it was actually sent
09:00Primary workflow becomes unavailableScope and cause of the failure
09:20Minimum intake service is available in the scenarioWhether a real team could reproduce this
09:50Primary application is available againWhether records and actions are reconciled

The scripted interval from outage to minimum service is twenty minutes, within the proposed sixty-minute target on paper. The export checkpoint is fifteen minutes before the outage. That age aligns with the proposed RPO interval, but it does not prove the data can actually be recovered to that point.

The application becomes available fifty minutes after the outage. Do not label that time “full recovery.” The scenario contains unresolved records and an uncertain send, so the conditions for resuming automation remain unmet.

A useful exercise asks participants what evidence they would request next. If they cannot locate an authoritative record for the 08:52 request, record the uncertainty and investigate. Do not declare it lost merely because it is absent from the export, and do not declare it recovered because someone remembers receiving it.

Reconcile before replaying actions

An external action with an unknown result needs investigation before retry. Search the provider's authoritative status or receipt using the original operation identifier where supported. Compare the intended action, destination, and payload with the observed record.

If a message was already sent, a second send is not recovery. If its status remains unknown, keep it in an exception queue and assign a decision owner. A timeout tells you that the caller did not receive a timely answer; it does not necessarily tell you whether the remote action occurred.

Preserve manual work created during the outage. In the exercise, MC-001 is a new intake request requiring reconciliation. Link it to its eventual system record and record who checked the match. Do not overwrite a newer source record with an older export simply because the export is easier to load.

The Part 9 local simulation demonstrates transactional receipt handling in one SQLite database. It does not establish that a remote email or booking service supports the same behavior. Recovery instructions must match the actual provider and the particular action.

Communicate the limitation customers need to know

An outage message should explain what service is available, what remains unconfirmed, and when the customer can expect an update if that time is known. Avoid inventing a restoration estimate from an internal target.

For example, an authorized employee might adapt this draft after checking the facts: “We have recorded your appointment request. Our scheduling system is unavailable, so the time is not yet confirmed. We will contact you after we can check availability.” The draft is not a sent message or a promise that a particular appointment is available.

Keep internal incident notes more detailed than customer notices. Staff need dependencies, action identifiers, and reconciliation decisions. Customers usually need a clear statement of what happens to their request. Review any notifications with special contractual or legal requirements through the appropriate owner.

Practice restoration as a separate activity

After the tabletop, plan a bounded technical recovery test using permitted test records and an isolated destination. Name the person executing it, the backup or export version, the capabilities being restored, and the criteria for success.

Check that the restored records are usable, not merely present. Verify identifiers, relationships, permissions, source versions, and selected pending actions. Record actual start and completion times and any missing information. A successful test of one export does not establish every future backup's integrity.

The companion observation fields remain blank because no restoration was performed here. The arithmetic checker validates the scenario's times and statuses only. It does not connect to a provider, restore business data, or measure actual recovery performance.

Resume deliberately and improve the plan

Before restarting automation, confirm that access is correct, current sources are available, uncertain actions are resolved or safely isolated, manual work is reconciled, and an accountable person has authorized the next scope. Consider resuming with a small reviewed batch and checking its results before increasing volume.

Record what the exercise or incident taught you. An unreachable backup contact, an unreadable export, or an ambiguous approval is a specific repair task. Give it an owner and verify the repair before closing it.

Finish with a runbook another person can use and a list of unproven recovery assumptions. A business becomes more dependable when it can maintain useful service and reconstruct trustworthy state. The final chapter uses that capability, together with quality and value evidence, to decide what should expand over the next ninety days.