Timo Dimitriadis, Lutz Duembgen, Alexander Henzi, Marius Puke, Johanna Ziegel
arXiv 8 Mar 2022 · Mathematics — Statistics Theory · publishedBiometrika (2022) · 16 citations (OpenAlex)
arXiv:2203.04065 · PDF · DOI · OpenAlex · Extracted main text
Probability predictions from binary regressions or machine learning methods ought to be calibrated: If an event is predicted to occur with probability $x$, it should materialize with approximately that frequency, which means that the so-called calibration curve $p(\cdot)$ should equal the identity, $p(x) = x$ for all $x$ in the unit interval. We propose honest calibration assessment based on novel confidence bands for the calibration curve, which are valid only subject to the natural assumption of isotonicity. Besides testing the classical goodness-of-fit null hypothesis of perfect calibration, our bands facilitate inverted goodness-of-fit tests whose rejection allows for the sought-after conclusion of a sufficiently well specified model. We show that our bands have a finite sample coverage guarantee, are narrower than existing approaches, and adapt to the local smoothness of the calibration curve $p$ and the local variance of the binary observations. In an application to model predictions of an infant having a low birth weight, the bounds give informative insights on model calibration.
appendix boundary found by appendix_command · 59% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Yang, F. and Barber, R. F (2019) Contraction and uniform convergence of isotonic regression | 0.896 | 25 | 8 | 72% |
| 2 | Dimitriadis, T., Gneiting, T., and Jordan, A. I (2021) Stable reliability diagrams for probabilistic classifiers self | 0.843 | 3 | 3 | 100% |
| 3 | Nattino, G., Finazzi, S., and Bertolini, G (2014) A new calibration test and a reappraisal of the calibration belt for the assessment of prediction models based on dichotomous ou… | 0.811 | 4 | 2 | 100% |
| 4 | R Core Team (2022) R: A language and environment for statistical computing | 0.644 | 2 | 2 | 100% |
| 5 | Hoeffding, W (1963) Probability inequalities for sums of bounded random variables | 0.511 | 2 | 2 | 50% |
| 6 | Koenker, R. and Yoon, J (2009) Parametric links for binary choice models: A Fisherian–Bayesian colloquy | 0.511 | 2 | 2 | 50% |
| 7 | Quinn, J.-A., Munoz, F. M., Gonik, B., Frau, L., Cutland, C., Mallet… (2016) Preterm birth: Case definition & guidelines for data collection, analysis, and presentation of immunisation safety data | 0.511 | 2 | 2 | 50% |
| 8 | Allison, P. J (2014) Measures of fit for logistic regression | 0.405 | 1 | 1 | 100% |
| 9 | Bertolini, G., D'Amico, R., Nardi, D., Tinazzi, A., and Apolone, G (2000) One model, several results: the paradox of the Hosmer-Lemeshow goodness-of-fit test for the logistic regression model | 0.405 | 1 | 1 | 100% |
| 10 | Clopper, C. J. and Pearson, E. S (1934) The use of confidence or fiducial limits illustrated in the case of the binomial | 0.405 | 1 | 1 | 100% |
Showing the top 10 of 31 scored citations.