Kentaro Hoffman, Stephen Salerno, Jeff Leek, Tyler McCormick
arXiv 22 May 2024 · Statistics — Methodology
arXiv:2405.13926 · PDF · DOI · OpenAlex · Extracted main text
Large-scale prediction models using tools from artificial intelligence (AI) or machine learning (ML) are increasingly common across a variety of industries and scientific domains. Despite their effectiveness, training AI and ML tools at scale can cost tens or hundreds of thousands of dollars (or more); and even after a model is trained, substantial resources must be invested to keep models up-to-date. This paper presents a decision-theoretic framework for deciding when to refit an AI/ML model when the goal is to perform unbiased statistical inference using partially AI/ML-generated data. Drawing on portfolio optimization theory, we treat the decision of {\it recalibrating} a model or statistical inference versus {\it refitting} the model as a choice between “investing” in one of two “assets.” One asset, recalibrating the model based on another model, is quick and relatively inexpensive but bears uncertainty from sampling and may not be robust to model drift. The other asset, {\it refitting} the model, is costly but removes the drift concern (though not statistical uncertainty from sampling). We present a framework for balancing these two potential investments while preserving statistical validity. We evaluate the framework using simulation and data on electricity usage and predicting flu trends.
appendix boundary found by none_found · 100% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Lazer, D., R. Kennedy, G. King, and A. Vespignani (2014) Google flu trends still appears sick: An evaluation of the 2013-2014 flu season | 0.644 | 4 | 1 | 100% |
| 2 | Angelopoulos, A. N., S. Bates, C. Fannjiang, M. I. Jordan, and T. Zr… (2023) Prediction-powered inference | 0.644 | 2 | 2 | 100% |
| 3 | Hoffman, K., S. Salerno, A. Afiaz, J. T. Leek, and T. H. McCormick (2024) Do we really even need data? self | 0.644 | 2 | 2 | 100% |
| 4 | Angelopoulos, A. N., J. C. Duchi, and T. Zrnic (2024) PPI++: Efficient prediction-powered inference | 0.585 | 3 | 1 | 100% |
| 5 | Fan, S., A. Visokay, K. Hoffman, S. Salerno, L. Liu, J. T. Leek, and… (2024) From narratives to numbers: Valid inference using language model predictions from verbal autopsy narratives self | 0.511 | 2 | 1 | 100% |
| 6 | Bonhomme, S. and M. Weidner (2022) Minimizing sensitivity to model misspecification | 0.511 | 2 | 1 | 100% |
| 7 | Wooldridge, J. M (2010) Econometric Analysis of Cross Section and Panel Data | 0.405 | 1 | 1 | 100% |
| 8 | Sumi, S. M (2021) Automated prediction of voter's party affiliation using ai | 0.405 | 1 | 1 | 100% |
| 9 | Quang, P. X (1985) Robust sequential testing | 0.405 | 1 | 1 | 100% |
| 10 | White, H (1982) Maximum likelihood estimation of misspecified models | 0.405 | 1 | 1 | 100% |
Showing the top 10 of 56 scored citations.