Evaluate Continuous Ranked Probability Score as a Forecasting Performance Measure

[antoinecarme]

Continuous Ranked Probability Score (CRPS) is a non-parametric (distribution-based) measure of the quality of a forecast.

As such, it should be interesting to evaluate it as the default performance measure used in PyAF for model selection (as a replacement for MAPE, which has some known issues with zero values in the signal).

Some references :

1. https://github.com/FK83/scoringRules
2. https://stats.stackexchange.com/questions/112250/understanding-the-rank-probability-score
3. https://www.kaggle.com/wiki/ContinuousRankedProbabilityScore
4. https://cran.r-project.org/web/packages/scoringRules/vignettes/gettingstarted.html
5. MAPE/sMAPE known issues : https://robjhyndman.com/hyndsight/smape/

[antoinecarme]

A similar method was used in scoring for GEFCom 2014 :

https://robjhyndman.com/hyndsight/gefcom2014/

[antoinecarme]

Probabilistic Forecasting (forecasting the signal distribution/quantiles over the horizon) :

1. Ichiro Takeuchi, Quoc V. Le, Timothy D. Sears, Alexander J. Smola, (2006), Nonparametric Quantile Estimation, Journal of Machine Learning Research 7 1231–1264
   https://www.lokad.com/probabilistic-forecasting-definition

2. Use and Communication of Probabilistic Forecasts Adrian E. Raftery University of Washington August 22, 2014
   https://arxiv.org/pdf/1408.4812.pdf

3. https://www.lokad.com/probabilistic-forecasting-definition

[antoinecarme]

Closing

[antoinecarme]

https://otexts.com/fpp3/distaccuracy.html