Why can't demand forecast error be measured with a single number?
Because which error matters depends on the decision the forecast feeds. Purchasing, production planning and shift planning look at different horizons, different levels of detail and different costs. For a store's weekly ordering decision, what matters is how accurately demand for that item in that store is forecast as far ahead as the supplier lead time. The error in a monthly category total does not answer that question; and as forecasts are aggregated, item errors cancel each other out and the picture looks better than it is.
So before you measure anything, write down four things: which decision the forecast feeds, the planning unit (for example store, item and week), the horizon (for example four weeks ahead, matching the lead time) and the cost of each direction of error. Under-forecasting leads to stockouts and lost sales; over-forecasting leads to holding costs, waste or markdowns. These two costs are rarely equal, and you need to know them to choose the right error measure.
Which error measure should you use, and when?
There is no single "best" measure; each one answers a different question and rewards different behaviour. The table below summarises the most common ones:
| Measure | What it tells you | Watch out for |
|---|---|---|
| MAE (mean absolute error) | Average deviation, in units | Scale-dependent; values for different items cannot be added up |
| RMSE (root mean squared error) | Penalises large errors more heavily | Sensitive to a few large errors |
| MAPE (mean absolute percentage error) | Easy to read as a percentage | Undefined for zero demand, explodes near zero |
| WAPE (total absolute error / total demand) | Volume-weighted percentage error | Fast sellers dominate; look by segment too |
| MASE (mean absolute scaled error) | Error relative to a simple naive forecast's error | Comparable across items; the team needs to learn how to read it |
| Mean bias (signed error) | Are you consistently over- or under-forecasting? | Absolute measures hide the direction; track it separately |
| Pinball (quantile) loss | Quality of forecast intervals and quantiles | For safety stock and service-level decisions |
There is an important detail behind these choices. According to the chapter on forecast accuracy by Hyndman and Athanasopoulos, a method that minimises MAE forecasts the median, while one that minimises RMSE forecasts the mean. The same chapter explains that percentage errors are undefined at zero, become extreme near zero and assume a meaningful zero, and that the symmetric MAPE (sMAPE) is not recommended. scikit-learn's regression metrics documentation also shows that MAPE can produce arbitrarily large values when true values are close to zero, and lists pinball loss as the consistent scoring function for quantile forecasts.
For slow-moving items this difference flows straight into stock decisions. The paper reporting the results of the M5 forecasting competition explains that many of its sales series contain lots of zeros, that absolute errors, being optimised for the median, would reward forecasts close to zero, and that because the goal was to forecast average sales a scaled measure based on squared errors (RMSSE) was chosen. The practical consequence: if you choose a model for slow movers by MAE alone, you can end up rewarding one that systematically orders too little. Hyndman and Koehler's paper on measures of forecast accuracy likewise shows that percentage measures cannot be used on intermittent demand data containing zeros, and proposes MASE as the standard measure for comparing accuracy across multiple series.
How should demand history be cleaned before measuring?
Sales data is not demand data; mark the difference before you measure. In a week when the item was out of stock, sales may be zero but demand was not; days when the store was closed, periods when the data feed broke and one-off bulk orders should be separated in the same way. Record promotions and price changes together with whether they were known at the time the forecast was made. Never give the model anything that was not known at forecast time, such as later-corrected sales totals or actual promotion results; otherwise the backtest will look better than production can ever be. An error calculated without this separation rewards a model in stockout weeks for "correctly forecasting zero demand". To assess more broadly whether your data is ready for an AI project, use the scorecard in our article on data readiness before AI.
Which baseline should a model be tested against?
Compare every model with simple methods first: a naive forecast that repeats the last observation, a seasonal naive forecast that uses the same period last year or last season, and a recent moving average. The chapter on simple forecasting methods notes that these methods are sometimes the best option available, and that in many cases they serve as benchmarks that any newly developed method has to beat. These baselines are not weak enough to dismiss, either: in the M5 competition, only about a third of more than 5,500 teams beat the seasonal naive forecast.
Make the comparison with a rolling-origin backtest rather than a single train-test split. Time series cross-validation uses only observations before each test point to forecast it, and it can be adapted to multi-step forecasts. Match the horizon to the decision horizon: if the order lead time is four weeks, measure the error of the four-weeks-ahead forecast; the one-week-ahead error can be misleadingly optimistic for this decision.
Hypothetical example: a baseline-versus-model worksheet
Consider weekly demand for an item a bakery chain bakes fresh every day, in one of its stores. Unsold items that week are waste, and unmet demand is a lost sale; a safety margin of 2 units is added every week. In week 2 demand really is zero, even though the store was open and the item was on the shelf. The figures are illustrative.
| Week | Actual demand | Seasonal naive forecast | Model forecast |
|---|---|---|---|
| 1 | 12 | 10 | 11 |
| 2 | 0 | 4 | 2 |
| 3 | 8 | 6 | 9 |
| 4 | 15 | 12 | 13 |
| 5 | 10 | 14 | 9 |
| 6 | 5 | 6 | 7 |
| Measure | Seasonal naive | Model |
|---|---|---|
| MAE (units) | 2.67 | 1.50 |
| WAPE | 32% | 18% |
| Mean bias (units) | +0.33 | +0.17 |
| MAPE | Undefined (week 2 is zero) | Undefined |
| Waste (units) | 15 | 13 |
| Lost sales (units) | 1 | 0 |
| Cost (lost sale 30 TRY, waste 10 TRY) | 180 TRY | 130 TRY |
The worksheet shows three things. First, a single zero week makes MAPE unusable. Second, an improvement of about 44% in MAE turns into a cost reduction of about 28%; lower error and money saved are not linear. Third, the decision should be made on cost: if a lost sale costs far more than waste, a model with a slightly worse average error that does not under-forecast can be the more profitable one. Six weeks and one item prove nothing; for a real comparison, repeat the same calculation across many items, stores and forecast origins.
At what level should error be reported?
Calculate the error first at the level where the decision is made, such as store, item and week, and then aggregate it with a deliberately chosen weight. An unweighted average gives an item that sells a few units a year the same importance as the best seller; weighting by units puts cheap fast sellers in front. Weighting by monetary value is often closer to the decision's impact; the M5 study also weighted series by their recent sales value when computing overall accuracy. Whichever weight you choose, show a segment table next to the total: a deterioration in slow movers barely shows up in the aggregate.
How do you evaluate forecast intervals and safety stock?
A point forecast cannot drive a safety-stock decision on its own; you also need to know how much uncertainty there is. If the model produces quantile or interval forecasts, measure two things: the pinball loss of the quantiles and the coverage rate. A 90% interval should contain the actual value about 90% of the time in the backtest. If coverage is clearly below that, safety stock will be too low and stockouts will rise; if it is well above, you are holding more stock than you need.
How do you monitor error in production?
Track error by segment rather than as one overall number: fast and slow movers, new items, new stores, promotion periods. Look at error separately when the range changes, when a new store opens and when the promotion calendar changes. Keep signed bias as its own indicator too; bias in the same direction for several periods in a row is a sign the model has missed a change. As a simple rule, if the sum of signed errors over the last eight periods approaches the sum of absolute errors over the same periods, meaning nearly all errors point the same way, review the model.
Let planners flag events the model does not know about, such as a supplier shutdown or a local event, and adjust the forecast with a stated reason. Store the original forecast, the adjustment, the reason and the actual demand together; after a few cycles, measure whether the adjustments really reduce error and cost. Do not assume that either the model or the planner is always right; evaluate both with the same measure.
Common mistakes
- Reporting MAPE for items with zero and intermittent demand.
- Including stockout weeks in the measurement as genuine zero demand.
- Putting a model into use without comparing it with a simple baseline.
- Measuring at a horizon shorter than the decision horizon and generalising the result.
- Reporting error only at the aggregate level and hiding the problem in slow movers.
- Declaring a winner without translating the error reduction into stock and cost outcomes.
Measuring error reliably depends on a data foundation that keeps sales, stock and promotion history with shared definitions and full history. A data warehouse, data quality rules and reporting dashboards are part of the scope described on our big data and data engineering service page.