Large-change forecast accuracy: Reanalysis of M3-Competition data using receiver operating characteristic analysis

Wilpen L. Gorr*, Matthew J. Schneider

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

8 Scopus citations


This paper applies receiver operating characteristic (ROC) analysis to micro-level, monthly time series from the M3-Competition. Forecasts from competing methods were used in binary decision rules to forecast exceptionally large declines in demand. Using the partial area under the ROC curve (PAUC) criterion as a forecast accuracy measure and paired-comparison testing via bootstrapping, we find that complex univariate methods (including Flores-Pearce 2, ForecastPRO, Automat ANN, Theta, and SmartFCS) perform best for this purpose. The Kendall tau test of dependency for PAUC and a judgmental index of forecast method complexity provide further confirming evidence. We also found that decision-rule combination forecasts using three top methods generally perform better than the component methods, although not statistically so. The top methods for forecasting large declines match the top methods for conventional forecast accuracy in the M3-Competition's micro monthly time series, and therefore, evidence from the M3-Competition suggests that practitioners should use complex univariate forecast methods for operations-level forecasting, for both ordinary and large-change forecasts.

Original languageEnglish (US)
Pages (from-to)274-281
Number of pages8
JournalInternational Journal of Forecasting
Issue number2
StatePublished - Apr 2013


  • Exceptions reporting
  • Forecasting
  • Large-change forecast accuracy
  • M3-Competition
  • ROC

ASJC Scopus subject areas

  • Business and International Management


Dive into the research topics of 'Large-change forecast accuracy: Reanalysis of M3-Competition data using receiver operating characteristic analysis'. Together they form a unique fingerprint.

Cite this