Run an AI Pilot and Decide Whether to Expand It

Compare completed work, inspect the failures, and make the decision you said you would make.

The pilot team celebrates because an AI assistant produces replies in seconds. Then a reviewer reports spending the afternoon checking policy statements and repairing missing details. The demonstration was fast; the completed workflow was less impressive.

Article 063 explained how to choose a promising pilot. This article takes the next step: evaluate its results. You need a fair comparison, full effort measurements, and a decision rule established before anyone sees the outcome.

A pilot can justify expansion, revision, or stopping. Each is a useful result when it follows the evidence.

Write the question so a result can answer it

For this lesson, imagine a fictional organization testing AI-assisted drafts of routine service replies. The question is: “Can the assisted process reduce completed handling effort while preserving factual accuracy and authorized commitments?”

The process includes preparing a draft, reviewing it, correcting it, and recording the final result. Excluded cases include account changes and policy exceptions. All replies remain subject to review during the pilot.

Before collecting results, the fictional team sets these criteria:

• At least a 20% reduction in handling time for comparable completed cases.
• No observed critical error, defined here as an unauthorized commitment or materially incorrect policy statement in a generated draft.
• A plausible recurring benefit after maintenance and tool costs.
• Sufficient evidence to describe what was tested and what remains uncertain.

These are exercise-specific criteria. Their strict treatment of draft errors is a deliberate local choice, even if review catches the error before release. The team can investigate and revise after a failure, but it cannot claim the original criterion was met.

Design a comparison that does not favor the result

Use representative work from the intended scope. Classify common case types in advance, then distribute similar work between approaches. If easy questions go to AI and difficult ones to the baseline, the comparison answers little.

Account for who performs the work and in which order. A person handling the same question twice may be faster the second time because they remember the answer. Parallel case sets, randomized assignment where practical, and a consistent review standard can reduce such problems.

Microsoft’s experimentation guidance emphasizes clear hypotheses and success metrics, along with measures that detect harmful changes and data-quality problems. The principle applies even when a small operational pilot cannot support a large statistical claim. Microsoft Research: Pre-Experiment Trustworthiness Patterns.

Record deviations. If a tool fails and the worker finishes manually, retain that case in the pilot record. Deleting failed attempts makes the assisted process look more reliable than the deployed workflow would be.

Inspect the synthetic dataset

The following data are invented for analysis practice. They represent equal counts of three case types in each approach. All listed cases were completed under the same fictional review standard. The table reports active labor, including review and rework, rather than elapsed turnaround time.

The AI handling total is 70 + 180 + 110 = 360 minutes. Compared with 480 baseline minutes, it saves 120 minutes, a 25% reduction in handling time. The raw speed criterion is met.

The eligibility error was an unauthorized promise in a draft, caught during review. No critical error reached a recipient in the synthetic record. That is evidence that review intercepted this error; it is not evidence that critical drafts never occurred.

Under the predefined rule, the zero-critical-draft-error criterion failed. The honest decision cannot be “all criteria passed.”

Add setup, maintenance, and cash costs

Suppose the fictional pilot also required 180 minutes of initial setup and 30 minutes of maintenance. Its total active effort becomes 360 + 180 + 30 = 570 minutes. For this pilot period, that is 90 minutes more than the 480-minute baseline.

Initial setup may be reusable, so show it separately from recurring effort. At the same volume and task mix, recurring assisted effort would be 360 + 30 = 390 minutes, a 90-minute saving compared with baseline. That is an 18.75% reduction after maintenance, before any additional support burden.

Assume a hypothetical tool cost of $20 for the period and labor valued at $30 per hour. Baseline labor is eight hours, valued at $240. Pilot labor is 9.5 hours, valued at $285, plus $20 for the tool: $305. Recurring labor at 6.5 hours would be $195, plus $20: $215.

The implied recurring capacity value is $25 per period, under those assumptions. It is not automatically a $25 cash saving or additional revenue. Setup labor valued at $90 would take 3.6 equivalent periods to recover at that rate, provided performance and volume remain stable. The failed quality criterion still prevents expansion under the current plan.

Look inside the average

The case types behave differently. Opening-hours handling falls from 120 to 70 minutes, a reduction of about 41.7%. Eligibility handling falls from 160 to 110, about 31.3%, but includes the critical error. Document requests fall only from 200 to 180, a 10% reduction.

The overall average hides both the attractive narrow use case and the problematic one. A reasonable next experiment might focus on opening-hours questions, provided the source can be kept current and the scope is clearly defined.

Do not describe a post hoc subgroup as if it had passed an independently designed trial. You noticed it after inspecting the results. Treat it as a reason to design another test, not as automatic permission to expand.

Also ask whether the workload mix will remain the same. If future work consists mostly of document requests, the overall 25% handling reduction will not transfer. A forecast needs the expected mix, not only the best-performing category.

Preserve uncertainty in the interpretation

This small dataset does not include worker-level variation, daily learning effects, or independent repeat trials. It therefore cannot establish how consistently the result will recur. Zero observed critical errors in one category does not prove zero underlying risk.

Research findings need similar care. METR’s early-2025 developer study challenged participants’ expectations about speed, while its February 2026 follow-up described substantial selection and measurement problems in later data. Those studies concern particular developer settings; they support the need for careful measurement, not a universal verdict about every AI workflow. METR’s early-2025 study, METR’s 2026 design update.

Record active labor and elapsed waiting separately when they matter. A workflow can free staff attention while taking longer to finish, or finish quickly while consuming extensive expert review. Which result matters depends on the service obligation.

Write a decision memo that follows the criteria

Here is an original example memo based on the synthetic dataset:

Decision: Revise; do not expand the current workflow. Handling time fell by 25% across 60 comparable cases per approach. However, the assisted process produced one critical draft error, failing the predefined quality condition. Setup and maintenance made total pilot effort higher than baseline, while the assumed recurring case suggests a modest capacity benefit. The service coordinator should investigate the eligibility error and design a separate, narrower pilot for opening-hours questions. All replies remain reviewed. No organization-wide productivity conclusion is supported by this dataset.

The memo names the reason for the decision, the owner, the next question, and the remaining boundary. It does not bury the failure under an enthusiastic headline.

A second pilot should use new cases, a current approved source, a fixed definition of critical error, and a representative workload. Keep a record of the prompt and tool configuration so a later change can trigger renewed evaluation.

For students: analyze evidence rather than sell the tool

A student team can evaluate this dataset without running a real customer-service experiment. Reproduce the totals, distinguish forecasts from observations, and explain the decision in plain language.

For a class or club pilot, agree on AI-use rules and record all attempts, including failures. If the exercise involves synthetic data only, do not write “our organization saved 25%.” Say “the synthetic dataset shows a 25% handling-time reduction before setup and maintenance.”

Practice: produce a decision memo

Calculate handling effort, total pilot effort, recurring effort, and the assumed cost comparison. Check each predefined criterion. Write a one-page expand, revise, or stop memo, then propose a second pilot addressing the largest uncertainty.

Completion check: The recommendation follows the stated criteria, includes review and rework, separates initial from recurring cost, and preserves the critical error. Another reader can reproduce every number and identify the next accountable decision.

For a stretch exercise, change the task mix so document requests dominate. Recalculate the forecast and explain whether the pilot’s apparent value survives.