Timothy Sudijono, Simon Ejdemyr, Apoorva Lal, Martin Tingley
arXiv 7 Dec 2024 · Statistics — Methodology · 1 citations (OpenAlex)
arXiv:2412.05508 · PDF · DOI · OpenAlex · Extracted main text
Experimentation in online digital platforms is used to inform decision making. Specifically, the goal of many experiments is to optimize a metric of interest. Null hypothesis statistical testing can be ill-suited to this task, as it is indifferent to the magnitude of effect sizes and opportunity costs. Given access to a pool of related past experiments, we discuss how experimentation practice should change when the goal is optimization. We survey the literature on empirical Bayes analyses of A/B test portfolios, and single out the A/B Testing Problem (Azevedo et al., 2020) as a starting point, which treats experimentation as a constrained optimization problem. We show that the framework can be solved with dynamic programming and implemented by appropriately tuning $p$-value thresholds. Furthermore, we develop several extensions of the A/B Testing Problem and discuss the implications of these results on experimentation programs in industry. For example, under no-cost assumptions, firms should be testing many more ideas, reducing test allocation sizes, and relaxing $p$-value thresholds away from $p = 0.05$.
appendix boundary found by none_found · 100% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Eduardo M Azevedo, Alex Deng, José Luis Montiel Olea, Justin Rao, an… (2020) A/b testing with fat tails | 1.000 | 24 | 8 | 100% |
| 2 | Eduardo M Azevedo, David Mao, José Luis Montiel Olea, and Amilcar Ve… (2023) The a/b testing problem with gaussian priors | 1.000 | 15 | 6 | 100% |
| 3 | David Goldberg and James E Johndrow (2017) A decision theoretic approach to a/b testing | 1.000 | 8 | 4 | 100% |
| 4 | F Richard Guo, James McQueen, and Thomas S Richardson (2020) Empirical bayes for large-scale randomized experiments: a spectral approach | 1.000 | 8 | 4 | 100% |
| 5 | Charles F Manski (2019) Treatment choice with trial data: Statistical decision theory should supplant hypothesis testing | 1.000 | 7 | 3 | 100% |
| 6 | Alex Deng (2015) Objective bayesian two sample hypothesis testing for online controlled experiments | 1.000 | 5 | 3 | 100% |
| 7 | Alberto Abadie, Anish Agarwal, Guido Imbens, Siwei Jia, James McQuee… (2023) Estimating the value of evidence-based decision making | 0.874 | 5 | 2 | 100% |
| 8 | Simon Ejdemyr, Martin Tingley, Yian Shang, and Travis Brooks (2024) Estimating the returns from an experimentation program self | 0.874 | 5 | 2 | 100% |
| 9 | Charles F Manski (2004) Statistical treatment rules for heterogeneous populations | 0.843 | 3 | 3 | 100% |
| 10 | Bradley Efron (2012) Large-scale inference: empirical Bayes methods for estimation, testing, and prediction, volume 1 | 0.644 | 2 | 2 | 100% |
Showing the top 10 of 58 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.
| Citing paper | Intensity | Mentions | Sections | |
|---|---|---|---|---|
| 1 | Compound Selection Decisions: An Almost SURE Approach | 1.000 | 5 | 3 |
| 2 | The Bias-Variance Tradeoff in Long-Term Experimentation | 0.644 | 2 | 2 |