Forecast Demand and Plan for Uncertainty

Test predictions using only information available when the forecast would have been made.

A forecasting model performs beautifully on a randomly shuffled dataset. Then it misses the next month.

One possible reason is that the evaluation allowed information from the future to influence the prediction. Another is that the model learned a pattern that no longer holds. A third is that the chosen accuracy measure did not reflect the decision.

Forecasting is about making a useful statement before the outcome is known. Its evaluation should recreate that information boundary as closely as the task requires.

AI can help organize the history, propose baselines, write evaluation code, and explain uncertainty. It cannot make a hindsight calculation become a forecast by giving it a confident title.

Define the target and the forecast origin

Our fictional example predicts weekly demand for a teaching product. Demand means requested units, including requests that could not be immediately fulfilled. The packet assumes this demand measure is fully recorded; it is not a sales series capped by available stock.

The decision is to plan resources for the next four weeks. The forecast is issued at the end of Week 8. All four predictions must therefore use information available by that time.

The first eight observed weekly values are:

10, 12, 11, 13, 12, 14, 13, 15 units.

This short history does not establish a reliable seasonal pattern. It is enough to teach a baseline and an evaluation calculation, not to justify a production model.

Record the target, horizon, update frequency, reporting calendar, and data-availability delay. A weekly value that is only finalized several days later was not necessarily available at the nominal end of that week.

Build a simple baseline

A last-value baseline predicts the most recently observed value. At the end of Week 8, that value is fifteen. Our fixed-origin four-week baseline therefore predicts:

15, 15, 15, 15.

The baseline is deliberately simple. Its purpose is to provide a clear reference that a more complex method must improve upon for the intended decision.

In another setting, a seasonal baseline might use the corresponding period from an earlier cycle. That requires a meaningful season and appropriate history. Do not invent a yearly pattern from a few weeks of observations.

Hyndman and Athanasopoulos explain evaluation on held-out observations and common forecast-error measures in Forecasting: Principles and Practice, point forecast accuracy. The arithmetic below is a separate synthetic example.

Freeze candidate predictions before evaluating them

The teaching packet also supplies candidate forecasts of:

16, 16, 17, 17.

These are invented candidate outputs for the exercise, not predictions from a trained model. The scenario stipulates that they were fixed at the end of Week 8. Their purpose is to show how a proposed forecasting method would be evaluated.

The subsequently revealed demand values for Weeks 9–12 are 16, 14, 18, and 17.

The mean absolute error is total absolute error divided by four: 1.75 units for the baseline and 0.75 units for the supplied candidate.

On this constructed four-week comparison, the candidate has smaller average absolute error. That is the supported conclusion. It does not establish that an AI model was trained or that the candidate will outperform the baseline in future periods.

Inspect error direction and size

Define signed error before reporting it. In this article, signed error means forecast minus actual, so a negative number indicates underprediction. Some references use the opposite sign; consistency matters more than choosing one universal convention.

The baseline’s signed errors are −1, +1, −3, and −2, averaging −1.25 units. The candidate’s are 0, +2, −1, and 0, averaging +0.25 units.

A mean signed error near zero can hide large errors in opposite directions. That is why it should accompany, rather than replace, a magnitude measure.

Root mean squared error gives larger errors more influence. In this example, it is approximately 1.94 units for the baseline and 1.12 for the candidate. The units remain demand units, not percentages.

Percentage errors need special care when actual demand is zero or close to zero. A familiar-looking percentage metric can become undefined or unstable. Choose the measure based on the target and decision, and show its formula.

Match the evaluation to the way forecasts are used

Our comparison uses one forecast origin and a four-week horizon. It would be incorrect to update the baseline using Week 9’s actual demand while claiming it was the unchanged forecast issued at Week 8.

A rolling one-week-ahead evaluation is also legitimate. It answers a different operational question: how well can we forecast the next week when we update after each new observation?

For that design, use only earlier observations at each origin, preserve the model-update rule, and evaluate the appropriate future step. The authors’ time-series cross-validation chapter explains rolling origins and the importance of keeping future observations out of forecast construction.

When comparing methods, give them the same available information and update opportunities. A complex model using revised future data should not be compared with a baseline restricted to the original snapshot.

Look for less obvious leakage

Future information can enter through preprocessing as well as model inputs. A normalization fitted to the whole series, a revised historical field, or a rolling calculation that accidentally includes the target period can compromise the evaluation.

External predictors also have publication schedules. Realized weather for next week is not the same as a weather forecast available today. Final economic figures may differ from the preliminary release available at the historical forecast origin.

Keep an availability timestamp where this matters. Ask: “Could the operator actually have known this value when making the decision?” If not, the evaluation should not treat it as an available predictor.

Preserve the forecast as issued. Replacing old predictions with updated ones destroys the record needed to assess real forecasting performance.

Express uncertainty without inventing calibration

A point forecast is one number. A scenario is a result under stated assumptions. A prediction interval aims to describe a range for a future observation under a method and coverage interpretation.

These are different outputs. Writing “likely range: 12–20” does not establish an 80% or 95% prediction interval. A coverage claim needs a documented method and appropriate evaluation.

The four test observations here are too limited to establish robust interval performance across realistic conditions. You can propose how to assess intervals over more forecast origins, including both coverage and width. A range that includes everything may have high coverage while being useless for planning.

For unusual events, keep scenarios clearly labeled. A temporary closure, a promotion, or a supplier interruption may require a conditional planning case rather than a false promise that the historical model anticipated it.

Connect error to the decision’s costs

Suppose, purely for teaching, each underpredicted unit has a three-unit cost and each overpredicted unit has a one-unit cost. Assume no inventory carryover between weeks so each weekly error is charged separately.

The baseline underpredicts by six units in total and overpredicts by one. Its simplified error cost is:

6 × 3 + 1 × 1 = 19 currency units.

The candidate underpredicts by one and overpredicts by two:

1 × 3 + 2 × 1 = 5 currency units.

This result depends on the fictional cost and no-carryover assumptions. Real inventory planning needs opening stock, lead times, minimum orders, shelf life, service requirements, and other constraints. A forecast is an input to that decision, not automatically an order quantity.

A method with slightly higher average error can sometimes support a better decision under asymmetric costs. Evaluate the operational policy as well as the forecast metric when the project reaches that stage.

Record overrides and exceptional periods

People may know about an event the model does not capture. Record the original forecast, override, reason, author, information available, and eventual outcome.

Do not overwrite the original prediction and later attribute the revised result to the model. Compare model-only and combined decisions transparently.

Inspect unusual periods separately when they matter, but do not remove them from the main evaluation merely because they worsen the score. Report the full scope and any additional subset analysis.

Define what triggers review: a data-feed change, repeated large errors, a new operating regime, or a revised target definition. Retraining is one possible response; repairing the source or changing the decision process may be more appropriate.

A reusable prompt

Design a time-ordered forecast evaluation for this target and decision. Specify the forecast origin, horizon, update rule, data-availability times, baseline, candidate method, and error measures. Identify leakage through predictors, revisions, and preprocessing. Preserve forecasts as issued. Separate point forecasts, conditional scenarios, and evaluated prediction intervals. Compare operational consequences using explicit cost and inventory assumptions, and do not describe supplied synthetic predictions as a trained model.

For students: keep the future covered

Give one student the first eight weeks and let another hold the final four. Write the prediction rule before revealing the outcomes. Then calculate the errors and discuss what the result does and does not establish.

Mathematics students can compare absolute and squared error. Business students can examine asymmetric costs. Computing students can implement rolling-origin evaluation while checking which observations are available at each step.

Do not keep trying new rules against the same four outcomes and continue calling them untouched test data. Once they guide model choice, they belong to development, and another independent evaluation is needed for a final claim.

Practice: compare the forecasts fairly

Reproduce the table’s absolute errors and mean absolute errors. Calculate signed errors using this article’s forecast-minus-actual convention and apply the fictional asymmetric costs.

Then describe how a rolling one-week-ahead evaluation would differ from the fixed four-week forecast. Identify at least three fields whose availability timestamps could matter in a real demand project.

Completion check: Baseline MAE is 1.75 units, candidate MAE is 0.75, mean signed errors are −1.25 and +0.25, and simplified costs are 19 and 5. No future observation is used before its availability, and no four-point result is presented as general forecasting reliability.

Stretch: Design an evaluation covering ordinary weeks and exceptional events. Report overall and event-specific performance, preserve all forecast origins, and explain how you would assess the coverage and usefulness of prediction intervals.