Compare Options with Statistics and Experiments

Measure the size of a difference, examine its uncertainty, and ask whether the comparison supports a causal claim.

A team changes an invitation and sees participation rise from 20% to 25%. The report announces a 25% improvement caused by the new wording.

The arithmetic may be correct. The causal claim may not be. Perhaps the second invitation went to a different audience during a different season. Perhaps the outcome was counted differently. Perhaps ordinary variation explains part of the difference.

AI can help calculate effects, critique a design, and organize an experiment plan. The central task is matching the conclusion to the comparison that was actually made.

Define the treatment, outcome, and population

Our fictional example compares two versions of a registration invitation. The outcome is whether an eligible recipient registers within seven days. Each recipient contributes one yes-or-no outcome.

Version A has 40 registrations among 200 recipients. Version B has 50 among 200. The seven-day follow-up is complete in both groups under the teaching assumptions.

If A was sent last month to previous participants and B this month to newcomers, the groups differ in more than wording. The observed association does not isolate the invitation’s effect.

Write the causal question explicitly: “Among the defined eligible recipients, how would registration change if they received B instead of A under the same study conditions?” That question concerns an intervention and a comparison population, not simply two percentages.

NIST’s experimental-design guidance emphasizes defining the objectives before selecting a design. See NIST’s discussion of experiment objectives.

Calculate absolute and relative effects

The observed rates are:

A: 40/200 = 20%.
B: 50/200 = 25%.

The absolute difference is five percentage points. The relative increase compared with A is:

(25% − 20%) ÷ 20% = 25%.

“Five percentage points” and “25% relative increase” describe the same observed difference using different scales. Report both when confusion is likely.

The observed count difference is ten registrations in these equally sized groups. If group sizes were different, comparing counts alone could be misleading.

Do not describe a five-percentage-point difference as a five-percent relative increase. Denominators matter in statistical communication just as they do in dashboards.

Explain why samples vary

Even under a well-designed comparison, a different set of recipients could produce different outcomes. Sampling uncertainty concerns this variation under a specified model or design.

For a transparent arithmetic illustration, assume independent binary outcomes within and between the two groups and use a conventional large-sample approximation for the difference in proportions. The estimated standard error is:

√[(0.20 × 0.80)/200 + (0.25 × 0.75)/200] ≈ 0.0417.

An approximate 95% interval for B minus A is:

0.05 ± 1.96 × 0.0417, or approximately −0.0317 to 0.1317.

Expressed in percentage points, that is roughly −3.2 to +13.2. The interval includes zero and includes effects in both directions. NIST’s comparison of two proportions provides a reference for the underlying large-sample comparison. The numerical interval here is an original calculation under the stated assumptions, not a universal method for every experiment.

For small samples, extreme rates, clustered assignment, repeated recipients, or other designs, choose an appropriate method. Do not ask AI to apply one formula automatically to every pair of percentages.

Interpret an interval without overstating it

Under its assumptions, the interval expresses uncertainty in the estimated difference. It does not establish that the invitation caused the observed difference in the historical comparison.

A confidence interval also is not a statement that 95% of individual recipients will experience an effect in that range. The target is a population-level difference under the defined analysis, not an individual prediction.

The frequentist 95% level describes the long-run coverage of the method under its assumptions. It is not a guarantee about this one interval, and it does not remove bias from missing outcomes or unsuitable group selection.

A wide interval can mean the available evidence is insufficient to distinguish effects that would matter operationally. “No statistically clear difference” is different from “the two options are equivalent.” Demonstrating practical equivalence requires a suitable design and a predefined range of negligible differences.

Design a fairer experiment

For a proposed randomized study, define eligibility before assignment. Randomly assign each recipient to A or B using a recorded procedure. Keep the invitation timing, follow-up window, and outcome definition comparable except for the intended treatment difference.

Randomization balances groups in expectation; it does not guarantee identical groups in a finite sample. Check implementation and relevant baseline characteristics without treating every chance imbalance as proof that randomization failed.

Prevent duplicate or conflicting assignments. If a household or team shares invitations in a way that could affect others, consider whether the randomization unit should be a group rather than an individual. The analysis must match that design.

Record non-delivery, missing outcomes, and any departures from the plan. Do not exclude participants after assignment simply because their results are inconvenient. Define the primary analysis and handling rules in advance.

Predefine the metric and stopping rule

Choose a primary outcome, follow-up period, analysis method, and stopping rule before examining results. Secondary outcomes can be useful, but label them and account for the scope of the investigation.

Repeatedly checking for a favorable result and stopping at the first attractive point changes the interpretation of ordinary fixed-sample calculations. A sequential design can address planned monitoring, but it requires an appropriate method rather than improvisation.

Likewise, testing many outcomes, audiences, and time windows and reporting only the best one makes the result appear stronger than the full exercise supports.

A planning record should identify who can change the design and how deviations will be reported. The purpose is to preserve the meaning of the evidence, not to prevent justified operational decisions.

Separate statistical and practical importance

A small p-value, when calculated appropriately, concerns compatibility with a specified null model. It does not measure the size of an effect, the probability that the business plan is correct, or the value of adopting the change.

A very small effect can be statistically detectable in a large study while having little practical value. A potentially useful effect can remain uncertain in a small study.

For the invitation example, consider the cost of producing and administering the new version, capacity to serve additional registrants, and the consequences of errors. Define a meaningful effect range before using the results to choose an action.

Do not turn the observed ten-registration difference into a guaranteed future gain. A decision may reasonably incorporate uncertainty, implementation cost, and reversibility, but it should state that reasoning rather than present the point estimate as a promise.

Write the conclusion for the design you have

For the historical comparison, a justified statement is:

“Observed registration was 20% for A and 25% for B, a five-percentage-point difference. The groups were exposed in different periods and may differ in relevant ways, so the comparison does not isolate the effect of wording. Under a simplified independent-outcome calculation, the uncertainty interval is wide. A planned randomized comparison would address the causal question more directly.”

For an actually completed randomized study, the conclusion would need its real protocol, results, missingness, and analysis. This article has not run an invitation experiment; it has calculated and critiqued a fictional example.

Keep the report’s verbs precise: proposed, assigned, observed, analyzed, and adopted describe different stages.

A reusable prompt

Evaluate whether this comparison supports the stated conclusion. Define the population, treatment, outcome, assignment unit, follow-up period, and comparison design. Calculate absolute and relative effects with denominators. Identify confounding, missing outcomes, repeated observations, uncertainty-method assumptions, selective reporting, and stopping issues. Separate statistical evidence from practical value. Draft a justified conclusion and an improved experiment plan without claiming an unperformed study was randomized or completed.

For students: critique before calculating significance

Use the fictional invitation counts and list three reasons the historical groups might differ apart from wording. Then design a classroom-scale paper simulation of random assignment without contacting actual recipients.

Statistics students can reproduce the standard-error calculation. Social-science students can examine eligibility and interference. Business students can define the smallest effect that would matter to a decision and explain why the point estimate is insufficient by itself.

Follow course requirements for any study involving people. The numerical exercise can be completed entirely with synthetic outcomes, so learning the method does not require collecting personal data.

Practice: repair the A/B report

Rewrite: “B caused a 25% gain, so we should switch immediately.” Include both effect scales, the complete counts, the approximate interval, and the limitations of the historical comparison.

Then produce a proposed randomized protocol with eligibility, assignment, primary outcome, follow-up, missing-data handling, and a predetermined stopping rule. Identify which sample-size assumptions would need specialist review before a consequential study.

Completion check: You calculate 20% and 25%, a five-percentage-point difference, a 25% relative increase, and an approximate interval of −3.2 to +13.2 percentage points under the stated assumptions. You do not convert association into causation or uncertainty into proof of no effect.

Stretch: Plan sample-size analysis using an explicit baseline rate, meaningful effect size, error-rate targets, assignment unit, and expected attrition. Explain which assumptions most strongly affect the required sample and why AI-generated sample sizes need verification.