Max H. Farrell, Malika Korganbekova, Sanjog Misra
arXiv 7 Sep 2026 · Econometrics
arXiv:2609.07633 · PDF · Extracted main text
A/B tests are standard in firm decision making. In the standard pipeline, experimental data is converted to a deployment decision by applying a t-test of the difference in means (the lift) and deploying the treatment if lift is positive and statistically significant. This common workflow answers the wrong question. We argue that firms need a decision rule for economic payoffs in the future deployment environment, not a test of equality in the experimental sample. We develop an ambiguity-averse decision framework in which each arm is evaluated by its ambiguity-penalized value over distributions close to the experimental outcome distribution. The resulting rule has a simple closed form thanks to the Donsker-Varadhan representation and it requires only the outcome data from a standard A/B test plus one interpretable parameter governing trust in the experiment. Our rule is thus no more difficult to implement than a t-test. A mean-variance approximation shows how the rule penalizes variability, while a connection to utility maximization shows it to be a certainty equivalent. We are able to perform a real-world evaluation of our proposed rule in the context of digital marketing using an archive of 552 advertising experiments from an anonymous US-based online platform. The proposed rule substantially reduces regret relative to conventional hypothesis testing. The results show that economically conservative, distribution-aware deployment rules can outperform statistical-significance rules in digital experimentation.
appendix boundary found by none_found · 100% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Stoye, Jörg (2012) Minimax regret treatment choice with covariates or with limited validity of experiments | 1.000 | 7 | 3 | 100% |
| 2 | Azevedo, Eduardo M and Deng, Alex and Montiel Olea, José L and Weyl,… (2019) Empirical bayes estimation of treatment effects with many a/b tests: An overview | 0.928 | 4 | 3 | 100% |
| 3 | Manski, Charles F (2019) Treatment choice with trial data: Statistical decision theory should supplant hypothesis testing | 0.811 | 4 | 2 | 100% |
| 4 | Feit, Elea McDonnell and Berman, Ron (2019) Test & roll: Profit-maximizing A/B tests | 0.737 | 3 | 2 | 100% |
| 5 | Goldberg, David and Johndrow, James E (2017) A decision theoretic approach to a/b testing | 0.737 | 3 | 2 | 100% |
| 6 | Tetenov, Aleksey (2016) An economic theory of statistical testing | 0.737 | 3 | 2 | 100% |
| 7 | Azevedo, Eduardo M and Mao, David and Olea, José Luis Montiel and Ve… (2023) The A/B testing problem with Gaussian priors | 0.644 | 2 | 2 | 100% |
| 8 | Joo, Joonhwi and Chiong, Khai X (2026) Getting the most out of A/B tests using the asymptotic minimax-regret criteria | 0.644 | 2 | 2 | 100% |
| 9 | Kuhn, Daniel and Shafiee, Soroosh and Wiesemann, Wolfram (2025) Distributionally robust optimization | 0.644 | 2 | 2 | 100% |
| 10 | Manski, Charles F (2011) Choosing treatment policies under ambiguity | 0.644 | 2 | 2 | 100% |
Showing the top 10 of 24 scored citations.