arXiv 4 May 2020 · Statistics — Methodology · 11 citations (OpenAlex)
arXiv:2005.01835 · PDF · DOI · OpenAlex · Extracted main text
I provide a unifying perspective on forecast evaluation, characterizing accurate forecasts of all types, from simple point to complete probabilistic forecasts, in terms of two fundamental underlying properties, autocalibration and resolution, which can be interpreted as describing a lack of systematic mistakes and a high information content. This "calibration-resolution principle" gives a new insight into the nature of forecasting and generalizes the famous sharpness principle by Gneiting et al. (2007) from probabilistic to all types of forecasts. It amongst others exposes the shortcomings of several widely used forecast evaluation methods. The principle is based on a fully general version of the Murphy decomposition of loss functions, which I provide. Special cases of this decomposition are well-known and widely used in meteorology. Besides using the decomposition in this new theoretical way, after having introduced it and the underlying properties in a proper theoretical framework, accompanied by an illustrative example, I also employ it in its classical sense as a forecast evaluation method as the meteorologists do: As such, it unveils the driving forces behind forecast errors and complements classical forecast evaluation methods. I discuss estimation of the decomposition via kernel regression and then apply it to popular economic forecasts. Analysis of mean forecasts from the US Survey of Professional Forecasters and quantile forecasts derived from Bank of England fan charts indeed yield interesting new insights and highlight the potential of the method.
appendix boundary found by appendix_command · 85% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Gneiting, T., F. Balabdaoui, and A. E. Raftery (2007) Probabilistic forecasts, calibration and sharpness | 1.000 | 20 | 5 | 100% |
| 2 | Tsyplakov, A (2011) Evaluating density forecasts: A comment | 1.000 | 8 | 3 | 100% |
| 3 | Gneiting, T. and A. E. Raftery (2007) Strictly proper scoring rules, prediction, and estimation | 1.000 | 6 | 4 | 100% |
| 4 | Elliott, G. and A. Timmermann (2016) Economic Forecasting | 0.928 | 4 | 3 | 100% |
| 5 | Wilks, D. S (2011) Statistical Methods in the Atmospheric Sciences, Volume 100 | 0.928 | 4 | 3 | 100% |
| 6 | Gneiting, T. and R. Ranjan (2013) Combining predictive distributions | 0.874 | 6 | 2 | 100% |
| 7 | Galbraith, J. W. and S. van Norden (2012) Assessing gross domestic product and inflation probability forecasts derived from bank of england fan charts | 0.843 | 3 | 3 | 100% |
| 8 | Hamill, T. M (2001) Interpretation of rank histograms for verifying ensemble forecasts | 0.843 | 3 | 3 | 100% |
| 9 | Holzmann, H. and M. Eulert (2014) The role of the information set for forecasting — with applications to risk management | 0.843 | 3 | 3 | 100% |
| 10 | Diebold, F. X., T. A. Gunther, and A. Tay (1998) Evaluating density forecasts with applications to financial risk management | 0.811 | 4 | 2 | 100% |
Showing the top 10 of 69 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.
| Citing paper | Intensity | Mentions | Sections | |
|---|---|---|---|---|
| 1 | Measurability of functionals and of ideal point forecasts | 0.405 | 1 | 1 |
| 2 | Score-based calibration testing for multivariate forecast distributions | 0.405 | 1 | 1 |
| 3 | Testing Quantile Forecast Optimality | 0.405 | 1 | 1 |