TY - JOUR
T1 - Large-change forecast accuracy
T2 - Reanalysis of M3-Competition data using receiver operating characteristic analysis
AU - Gorr, Wilpen L.
AU - Schneider, Matthew J.
PY - 2013/4
Y1 - 2013/4
N2 - This paper applies receiver operating characteristic (ROC) analysis to micro-level, monthly time series from the M3-Competition. Forecasts from competing methods were used in binary decision rules to forecast exceptionally large declines in demand. Using the partial area under the ROC curve (PAUC) criterion as a forecast accuracy measure and paired-comparison testing via bootstrapping, we find that complex univariate methods (including Flores-Pearce 2, ForecastPRO, Automat ANN, Theta, and SmartFCS) perform best for this purpose. The Kendall tau test of dependency for PAUC and a judgmental index of forecast method complexity provide further confirming evidence. We also found that decision-rule combination forecasts using three top methods generally perform better than the component methods, although not statistically so. The top methods for forecasting large declines match the top methods for conventional forecast accuracy in the M3-Competition's micro monthly time series, and therefore, evidence from the M3-Competition suggests that practitioners should use complex univariate forecast methods for operations-level forecasting, for both ordinary and large-change forecasts.
AB - This paper applies receiver operating characteristic (ROC) analysis to micro-level, monthly time series from the M3-Competition. Forecasts from competing methods were used in binary decision rules to forecast exceptionally large declines in demand. Using the partial area under the ROC curve (PAUC) criterion as a forecast accuracy measure and paired-comparison testing via bootstrapping, we find that complex univariate methods (including Flores-Pearce 2, ForecastPRO, Automat ANN, Theta, and SmartFCS) perform best for this purpose. The Kendall tau test of dependency for PAUC and a judgmental index of forecast method complexity provide further confirming evidence. We also found that decision-rule combination forecasts using three top methods generally perform better than the component methods, although not statistically so. The top methods for forecasting large declines match the top methods for conventional forecast accuracy in the M3-Competition's micro monthly time series, and therefore, evidence from the M3-Competition suggests that practitioners should use complex univariate forecast methods for operations-level forecasting, for both ordinary and large-change forecasts.
KW - Exceptions reporting
KW - Forecasting
KW - Large-change forecast accuracy
KW - M3-Competition
KW - ROC
UR - http://www.scopus.com/inward/record.url?scp=84872701833&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84872701833&partnerID=8YFLogxK
U2 - 10.1016/j.ijforecast.2012.12.001
DO - 10.1016/j.ijforecast.2012.12.001
M3 - Article
AN - SCOPUS:84872701833
SN - 0169-2070
VL - 29
SP - 274
EP - 281
JO - International Journal of Forecasting
JF - International Journal of Forecasting
IS - 2
ER -