Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
50,323 characters · 17 sections · 20 citation commands
Towards Optimal Estimators for Randomized Control Trials
\def\spacingset#1{ {#1}} \spacingset{1}
\if11 \fi
\if01 {
} \fi
{\it Keywords:} experiments, data integration, causal inference, machine learning
\spacingset{1.8}
Randomized controlled trials (RCTs) have become the gold standard for causal inference across diverse scientific domains. The appeal of RCTs lies in their exceptional internal validity: treatment randomization effectively eliminates unobserved confounding, ensuring unbiased estimation of causal effects is possible. The knowledge from these experiments serves dual purposes---advancing scientific understanding of underlying causal mechanisms and informing critical downstream decisions about intervention deployment. Each experiment requires an estimation strategy that translates the observed outcomes into an estimate of the treatment effect, and currently there is no single strategy for selecting such an estimator for a class of related RCTs.
A foundational estimator for RCTs is the the standard difference-in-means (DM) estimator, which simply contrasts the average outcomes among those assigned to treatment and control in the experiment. While this estimator is unbiased it can suffer from inadequate precision. This limitation becomes particularly acute when treatment effects exhibit substantial heterogeneity or when outcome distributions display heavy-tailed characteristics---scenarios increasingly common in modern applications. The precision challenge has motivated extensive methodological development along two primary axes: covariate adjustment strategies and variance reduction techniques. The landscape of available methods has expanded dramatically in both directions due to advances in machine learning and causal inference, encompassing approaches from simple ordinary least squares to sophisticated random forests and neural networks to changes in the targeted estimand.
This methodological abundance naturally raises a fundamental question: which estimator should researchers choose for their analysis? Validation studies in causal inference have consistently demonstrated that no universal “optimal” estimator exists across all datasets. Selecting estimators on a case-by-case basis, however, introduces significant risks to scientific validity, as researchers may gravitationally select methods that support preferred narratives rather than those that provide the most reliable inference. We address this challenge by shifting focus from finding optimal estimators for individual RCTs to identifying superior approaches for entire families of related trials.
Within each scientific domain, hundreds or thousands of experiments are conducted each year with each experiment measuring the impact of potentially different treatments on the same outcome: e.g. technology companies like Amazon, Meta, and Google optimizing user experiences where the outcomes are click-through rates, pharmaceutical giants such as Eli Lilly and Pfizer evaluating drug efficacy in terms of all-cause-mortality and federal agencies including the FDA and NIH informing policy decisions --- where each experiment aims to evaluate the impact on the same outcome.
Our contribution is a general framework that characterizes optimal estimators for specified classes of RCTs, explicitly accounting for intended analytical goals and outcome characteristics. The framework employs sample splitting methodology to estimate evaluation metric distributions across trial families, ensuring valid asymptotic behavior while enabling systematic estimator comparisons. We demonstrate the framework's utility through two case studies: Amazon's Supply Chain Optimization Technology (SCOT) trials comprising a sample of 556 experiments RCTs focused on a common financial outcome metric, and the Strengthening Democracy Challenge dataset encompassing 25 interventions targeting political attitudes and behaviors voelkel2024mega. Our empirical findings confirm the absence of universally optimal estimators while providing actionable guidance for method selection. For the Amazon SCOT trials, we find that analytical objectives critically determine optimal choices. Similarly, analysis of the Democracy Challenge data reveals outcome-dependent optimal estimators, reinforcing the importance of context-specific method selection.
This work makes several key contributions to the statistical literature on causal inference. First, we provide a principled alternative to ad hoc estimator selection that preserves scientific integrity while accommodating methodological flexibility. Second, we demonstrate how evaluation criteria should align with analytical objectives, distinguishing between inference and decision-making goals. Third, we offer a practical framework that practitioners can readily implement across diverse experimental contexts. Finally, our empirical applications showcase the framework's value in both academic and business settings, highlighting its broad applicability across research domains.
Section (ref) discusses about relevant literature. Section (ref) introduces the setup and notation as well as discusses some of the existing estimator for treatment effects. Section (ref) introduces our framework and method for characterizing optimal estimator. Section (ref) discusses our business case study on Amazon data and Section (ref) discusses our social science case study on SDC data.
Randomized controlled trials have long been considered the gold standard for estimating causal effects across various disciplines imbens2015causal. The fundamental appeal of RCTs lies in their ability to balance confounders across treatment conditions via treatment randomization, thereby enabling unbiased estimation of average treatment effects. The simplest approach is the difference-in-means estimator, which directly compares outcome averages between treatment and control groups athey2017econometrics. While unbiased, researchers have increasingly recognized that precision can be substantially improved through more sophisticated methods without compromising internal validity Benkeser2020.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{Covariate Adjustment in RCTs.} Covariate adjustment has become standard for improving precision in RCT analysis. Incorporating pre-treatment covariates through regression models can significantly reduce variance in treatment effect estimates freedman2008regression, lin2013agnostic. Unlike in observational studies, the primary motivation here is variance reduction rather than addressing confounding tsiatis2008covariate. yang2001efficiency showed that ANCOVA estimators remain consistent even under model misspecification while providing efficiency gains. Recent work has explored high-dimensional regression adjustment using regularization methods such as LASSO bloniarz2016lasso and ridge regression wager2016high.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{Methods from Observational Studies Applied to RCTs.} Several methods developed for observational studies have been adapted to improve precision in RCTs. Inverse Probability Weighting (IPW) estimators account for chance imbalances in covariates hirano2003efficient, while meta-learners including T-learners, S-learners, and X-learners model treatment effect heterogeneity using machine learning approaches kunzel2019metalearners. Doubly robust estimators like Augmented IPW (AIPW) combine outcome regression with propensity score methods funk2011doubly, with recent extensions to double/debiased machine learning chernozhukov2018double.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{The Challenge of Method Validation and Selection.} With this proliferation of methods, the challenge has shifted from developing new estimators to selecting among existing alternatives. Traditional validation approaches for causal inference methods fall into three categories: face-validity tests that assess estimates against expert intuition, placebo or negative control tests that evaluate a method's ability to recover null effects under domain-driven constraints, and synthetic data studies with known data-generating mechanisms.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{Synthetic Data Approaches and Their Limitations.} Recent literature has explored sophisticated synthetic data generation to identify optimal estimators for particular contexts. Approaches using variational autoencoders (VAEs) and generative adversarial networks (GANs) have been proposed to simulate datasets with known causal structures athey2018gan, parikh2022validating. Particularly notable is the work by parikh2022validating, who developed a comprehensive framework for validating causal inference methods using VAE-based synthetic data generation, enabling systematic comparison across varying degrees of confounding and treatment effect heterogeneity.
Despite these advances, synthetic data approaches face important limitations that motivate our alternative framework. As noted by parikh2022validating and gentzel2019case, performance is highly sensitive to training procedures and hyperparameter choices of generative models. losch2021optimal found significant divergence between estimator rankings on synthetic versus real-world data, suggesting that synthetic performance may not translate reliably to actual applications. Moreover, the lack of theoretical guarantees regarding synthetic-to-real performance relationships creates the risk of “convenient estimator choice,” where researchers might selectively report results from synthetic settings favorable to preferred methods.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{Gap in Current Literature.} While substantial progress has been made in developing precision-enhancing methods and validation approaches, a critical gap remains in providing principled guidance for estimator selection in practice. Current approaches typically focus on either developing new methods or evaluating methods in isolation, rather than providing systematic frameworks for choosing among competing approaches based on research context and analytical objectives. Our work addresses this gap by proposing a framework that leverages families of related RCTs to identify optimal estimators while accounting for both intended use cases (inference versus decision-making) and distributional properties of outcomes within specific research domains. Unlike synthetic approaches, our framework directly evaluates estimators on collections of real RCTs sharing common outcome measures, providing stronger assurance regarding cross-context generalizability while maintaining principled statistical foundations.
We consider a framework where both individual studies and their design characteristics are drawn from broader populations, allowing us to develop estimator selection principles that generalize across families of related experiments.
Let $\mathcal{O}$ denote a countably infinite superpopulation of units eligible for treatment interventions, such as all potential customers, products, or individuals in a given domain. We consider a finite set of potential treatments $\mathcal{T} = \{0, 1, \ldots, K-1\}$, where treatment $0$ typically represents a control or baseline condition, and the remaining treatments represent various interventions of interest.
Our analysis focuses on a collection of $N$ studies, indexed by the random variable $S \in \{1, 2, \ldots, N\}$. Each study $S = s$ is generated through the following two-stage sampling process:
Stage 1: Treatment Selection. For study $s$, we first randomly sample two treatments $t_a^{(s)}$ and $t_b^{(s)}$ uniformly at random from $\mathcal{T} \times \mathcal{T}$. These represent the two treatment conditions to be compared in the study, where typically one serves as the active treatment and the other as a control or comparison condition.
Stage 2: Population Sampling. Given the selected treatment pair $(t_a^{(s)}, t_b^{(s)})$, we recruit $m_s$ units from the superpopulation $\mathcal{O}$ through some sampling mechanism (which may be random or non-random) to form the study sample $\mathcal{O}_s \subseteq \mathcal{O}$. The sampling mechanism may depend on study-specific factors such as eligibility criteria, temporal constraints, or resource limitations.
Thus, each study $s$ is uniquely identified by the triple $(t_a^{(s)}, t_b^{(s)}, \mathcal{O}_s)$, encompassing both the treatments under comparison and the specific population recruited for that comparison.
The estimand of interest for study $s$ is the average treatment effect within the study population:
where $Y_i(t)$ denotes the potential outcome for unit $i$ under treatment $t$, following the standard potential outcomes framework neyman1923application, rubin1974estimating.
For each unit $i \in \mathcal{O}_s$ recruited into study $s$, we observe the following data:
The observed dataset for study $s$ is $\mathcal{D}_s = \{(Y_i, T_i, X_i) : i \in \mathcal{O}_s\}$. Our framework leverages the collection of datasets $\{\mathcal{D}_s\}_{s=1}^N$ from this family of related studies to identify optimal estimators for the average treatment effects $\{\tau^{(s)}\}_{s=1}^N$.
We make the following key assumptions that enable our analysis:
Assumption (ref) ensures that studies differ only in their covariate distributions, not in the underlying causal mechanisms.
Under Assumption (ref), the average treatment effect in study $s$ can be identified as:
Assumption (ref) is crucial for our framework as it allows us to meaningfully compare estimator performance across studies---it ensures that any differences in estimator performance across studies can be attributed to differences in covariate distributions and sample compositions rather than fundamental differences in causal mechanisms. Assumption (ref) simplifies the statistical analysis by ensuring independence across study samples and avoiding complications from repeated measurements of the same units.
This setup naturally accommodates scenarios where studies share common treatments (when the same treatment pairs are drawn multiple times) as well as cases where studies evaluate entirely different treatment comparisons, providing a flexible framework for analyzing families of related experiments.
This section presents our methodology for rigorously assessing and comparing treatment effect estimators across families of randomized controlled trials. Our approach leverages the collection of studies to estimate the distribution of estimator performance, enabling principled comparisons while avoiding overfitting to any single dataset.
Let $\hat{\tau}^{(s)}_\alpha$ denote the estimated treatment effect when applying estimator $\alpha$ to study $s$, where the specific treatments $(t_a^{(s)}, t_b^{(s)})$ and sample $\mathcal{O}_s$ are determined by the study design. We assess estimator performance using metrics $\Delta(\cdot, \cdot)$ that quantify the discrepancy between estimates and true treatment effects, with the choice of metric depending on the intended use of the estimates.
Inference-Focused Applications. For applications where the primary goal is accurate parameter estimation, we use mean squared error:
This metric captures both bias and variance, providing a comprehensive measure of estimator precision for statistical inference.
Decision-Making Applications. For applications where estimates inform treatment rollout decisions, we focus on the economic consequences of estimation errors. We consider a simple decision rule where treatment is rolled out if $\hat{\tau}^{(s)}_\alpha > c \cdot \text{SE}(\hat{\tau}^{(s)}_\alpha)$ for some critical threshold $c \geq 0$. The regret from suboptimal decisions is:
where $|\mathcal{O}_s|$ represents the scale of potential implementation. This metric quantifies the opportunity cost when beneficial treatments are not rolled out (false negatives) and the harm when ineffective treatments are implemented (false positives).
For a given performance metric $\Delta$, we define the expected performance of estimator $\alpha$ in study $s$ as:
where the expectation is taken over the randomness in sample composition $\mathcal{O}_s$ and treatment assignment within the study.
The overall performance of estimator $\alpha$ across the family of studies is:
We say that estimator $\alpha$ dominates estimator $\beta$ if $M_\alpha < M_\beta$. To test this comparison, we define the test statistic:
where $\theta(\alpha,\beta) > 0$ indicates that $\alpha$ outperforms $\beta$.
Since true treatment effects $\{\tau^{(s)}\}_{s=1}^N$ are unknown, we cannot directly compute the performance metrics. We propose a sample-splitting approach that provides valid estimates of $\theta(\alpha,\beta)$ while avoiding overfitting.
Cross-Fitting Protocol. For each study $s$, we randomly partition the sample $\mathcal{O}_s$ into $K$ non-overlapping subsets of approximately equal size: $\{\mathcal{O}_{s,k}\}_{k=1}^K$ such that $\cup_{k=1}^K \mathcal{O}_{s,k} = \mathcal{O}_s$ and $\mathcal{O}_{s,i} \cap \mathcal{O}_{s,j} = \emptyset$ for $i \neq j$.
For each fold $k$, we:
The cross-fitted performance estimate for estimator $\alpha$ in study $s$ is:
This procedure leverages the fact that under Assumption (ref), the difference-in-means estimator is unbiased: $\mathbb{E}[\hat{\tau}^{(s)}_{\text{DM},k}] = \tau^{(s)}$. The cross-fitting ensures that the "ground truth" estimate and the estimator being evaluated use independent data, preventing overfitting.
The estimated overall performance is $\hat{M}_\alpha = \frac{1}{N} \sum_{s=1}^N \hat{\epsilon}^{(s)}_\alpha$, and the test statistic becomes:
The bound $B^{(s)}_\gamma$ has an intuitive interpretation: (i) the first term captures the cost of sign errors when $\hat{\tau}^{(s)}_{\text{DM},k}$ and $\tau^{(s)}$ have opposite signs, and (ii) the second term captures the cost of magnitude errors in the DM estimator.
Establishing asymptotic normality for $\hat{\theta}(\alpha,\beta)$ faces several complications: the asymptotic behavior depends critically on the smoothness of the performance metric $\Delta$ (our regret metric involves indicator functions that create discontinuities), our cross-fitting procedure introduces two-stage estimation complexity, and study heterogeneity in sample sizes and effect magnitudes can lead to non-identically distributed performance differences requiring specialized limit theorems. Additionally, multiple comparisons across estimators require family-wise error rate corrections that further complicate CLT-based inference.
Advantages of Permutation-Based Inference. Permutation tests elegantly sidestep these technical challenges by leveraging the fundamental exchangeability property under the null hypothesis $H_0: \theta(\alpha,\beta) = 0$, where estimator labels can be randomly reassigned without changing the distribution of performance differences.
Permutation Test Framework. Under the null hypothesis $H_0: \theta(\alpha,\beta) = 0$ (equal performance), estimators $\alpha$ and $\beta$ have identical expected performance across studies. This implies that the assignment of estimator labels to the observed performance differences is exchangeable under $H_0$.
Our test procedure works as follows:
While our cross-fitting approach provides consistent identification of dominant estimators asymptotically, finite-sample performance may be sensitive to outlier studies with unusually large outcome variability. Such high variability can arise from implementation issues, design problems, or genuinely different populations that may not be representative of the broader family of studies under consideration. To address this concern while maintaining statistical rigor, we extend our framework to incorporate study-specific weights that reflect the reliability and representativeness of each study's contribution to our estimator comparison.
We modify our overall performance metric to $M_{\alpha,w} = \mathbb{E}_S[w^{(s)} \cdot \epsilon^{(s)}_\alpha]$, where $w^{(s)} \geq 0$ is a study-specific weight. A natural weighting scheme assigns higher weight to studies with more precise treatment effect estimates through inverse-variance weighting: $w^{(s)} = (\text{Var}(\hat{\tau}^{(s)}_{\text{DM}}))^{-1}$. This approach has several desirable properties: studies with larger samples receive higher weight reflecting their greater statistical power, studies with lower outcome variance receive higher weight as they provide more precise estimates, and studies with severe implementation issues reflected in extremely high variance are automatically down-weighted.
To implement weighted comparisons within our cross-fitting framework, we estimate the weights using the full data in each study. For study $s$, we estimate $\hat{w}^{(s)}$ using the sample variance. The weighted cross-fitted performance estimate becomes $\hat{\epsilon}^{(s)}_{\alpha,w} = \frac{\hat{w}^{(s)}}{K} \sum_{k=1}^K \Delta(\hat{\tau}^{(s)}_{\alpha,-k}, \hat{\tau}^{(s)}_{\text{DM},k})$, and the weighted test statistic is $\hat{\theta}_w(\alpha,\beta) = \frac{1}{N} \sum_{s=1}^N (\hat{\epsilon}^{(s)}_{\beta,w} - \hat{\epsilon}^{(s)}_{\alpha,w})$.
The weighted approach provides a principled middle ground between purely rank-based comparison (counting only the number of studies where one estimator outperforms another, which ignores magnitude of performance differences) and unweighted comparison (treating all studies equally, which can be dominated by high-variance outliers). The inverse-variance weighting naturally balances these concerns by giving appropriate weight to each study's contribution based on its precision, while still incorporating information from all studies in the family.
We demonstrate our framework through analysis of randomized controlled trials conducted at Amazon to evaluate supply chain optimization policies. These experiments provide an ideal testbed for comparing treatment effect estimators across a large family of related studies with shared outcome measures and analytical objectives.
Our analysis leverages a sample of 556 RCTs conducted between 2016 and 2025 across diverse geographies, marketplaces, and product categories. Most trials compare a single treatment against control; for multi-arm trials, we construct pairwise comparisons between each treatment and control. While there is a diversity of treatments under study, the common theme for them is that they implement a change to Amazon’s supply chain network. Importantly, these changes are evaluated independently through randomized experiments. The data considered in this study include RCTs with independent units and treatments, satisfying all of the requirements of the methodology presented in this paper. For outcome variable we focus on a common financial outcome. However, the method is readily applicable for any outcome variable for MSE estimation and for any profit-related metric in terms of our regret derivations.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{Estimators.} We evaluate four base estimators: (1) Difference of Means (DM), (2) Ordinary Least Squares with covariate adjustment (OLS), (3) Linear T-learner with separate models per treatment arm, and (4) Weighted Least Squares (WLS) with weights $w_i = 1/(Y_{\text{pre},i} - \bar{Y}_{\text{pre}})^{2}$ that emphasize units near the pre-treatment mean. Given the heavy-tailed nature of this financial metric, we combine each base estimator with winsorization. Particularly, we consider following seven levels of winsorization: $p \in \{0, 0.001, 0.005, 0.01, 0.025, 0.05, 0.1\}$. We denote combinations as Estimator_Level (e.g., "DM_0.01" for difference-of-means with 1% winsorization). This yields 28 total estimator variants.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{Evaluation Metrics and Objectives.} Amazon's experimentation serves two objectives: (i) launch decisions and (ii) uplift measurement. For launch decisions, we use the regret metric $\Delta_{\text{Regret}}$ to identify estimators minimizing opportunity costs from false positives (launching ineffective policies) and false negatives (missing beneficial policies). For uplift measurement, we use Mean Squared Error ($\Delta_{\text{MSE}}$) to prioritize accurate and precise effect estimation. This dual evaluation demonstrates how optimal estimation strategies depend on analytical objectives.
Our analysis reveals substantial differences in estimator performance across evaluation objectives, confirming that optimal strategies are context-dependent. Figures (ref) and (ref) present results for MSE and regret-based evaluation.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{MSE-Based Performance for Inference.} For mean squared error (Figure (ref)), Linear T-learner with 0.5% winsorization (Linear-T_0.005) achieves minimum MSE. However, optimality varies by winsorization level: at levels below 0.5%, OLS becomes optimal. MSE displays the expected smooth, convex shape with a global minimum reflecting the bias-variance tradeoff. As winsorization increases, performance differences between DM, OLS, and Linear T-learner diminish, suggesting that winsorization's variance reduction eliminates the advantage of covariate adjustment -- though at the cost of increased bias.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{Regret-Based Performance for Decision-Making.} Regret-based evaluation\footnote{We normalize regret so that the value at critical threshold 1.96 for DM_0 equals 1.} (Figure (ref)) reveals a strikingly different ranking. Difference-of-means without winsorization (DM_0) achieves best performance, while Linear-T_0.005, the estimator with minimum MSE, performs suboptimally here.
This reversal illuminates a fundamental insight: estimators minimizing estimator error need not minimize decision costs. The DM estimator's unbiased, symmetric sampling distribution better aligns with minimizing decision errors despite higher variance. Linear-T_0 ranks second for regret, with OLS a close third, suggesting these covariate-adjusted methods offer reasonable compromises when objectives are mixed or uncertain.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{Practical Implications.} These results highlight that principled estimator selection requires explicit consideration of analytical objectives. For Amazon's context, we recommend DM_0 for launch decisions and Linear-T_0.005 for uplift measurement, with T-Learner_0 as a reasonable default when objectives are uncertain. The performance reversals between metrics underscore why ad hoc estimator selection may yield suboptimal outcomes in practice.
Next, we apply our framework to the publicly available Strengthening Democracy Challenge (SDC) data. Here, the primary goal is inferential: understanding intervention effects on democratic attitudes. We use mean squared error (MSE) as our evaluation metric ($\Delta$).
The SDC provides an ideal testbed featuring 25 interventions targeting similar outcomes with consistent baseline covariates voelkel2024mega. The experiment recruited 32,059 U.S. adults in January 2022 via an online platform, randomly assigning approximately 1,200 participants to each of 25 treatment conditions or control. Interventions ranged from perspective-taking exercises to media literacy training, each designed to be brief (2-8 minutes) and scalable. The control condition exposed participants to an unrelated article.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{Outcome Measures.} We focus on two constructs: Support for Partisan Violence (SPV) measured through three 7-point scale items ($SPV_1$: threatening messages, $SPV_2$: online harassment), and Partisan Animosity (PA) measured through feeling thermometers toward Republicans ($PA\text{-}Fth_{Rep}$) plus a Dictator Game allocation ($PA_{DG}$). Each intervention-control comparison constitutes a separate RCT, yielding 25 RCTs per outcome for systematic estimator evaluation.
Our analysis reveals substantial variation in optimal estimators across outcomes and evaluation metrics, reinforcing that estimator selection must align with both data characteristics and analytical objectives.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{MSE-Based Performance for Inference.} Figure (ref) displays mean squared error across estimators and winsorization levels. For $SPV_1$ and $SPV_2$, all estimators achieve best performance at high winsorization levels (around 10%), with OLS_0.1 emerging as optimal. Conditional on winsorization level, estimators perform nearly identically, suggesting weak associations between outcomes and pre-treatment features, low treatment effect heterogeneity, and heavy-tailed distributions characteristic of social science outcomes.
In contrast, $PA\text{-}Fth_{Rep}$ shows difference-in-means performing substantially worse than covariate-adjusted estimators (IPW, OLS, linear T-learner), with winsorization providing minimal benefit even at 10%. This pattern indicates stronger covariate-outcome relationships where adjustment methods yield precision gains. For $PA_{DG}$, difference-in-means performs comparably to other estimators, but substantial winsorization (5-10%) significantly improves performance across all methods, revealing outcome-specific sensitivity to extreme values.
These patterns demonstrate that optimal estimators and hyperparameters—including winsorization levels—depend critically on outcome characteristics. Heavy-tailed outcomes with weak covariate relationships benefit from aggressive winsorization, while outcomes with strong covariate associations gain more from adjustment methods than from trimming extremes.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{Regret-Based Performance for Decision-Making.} When evaluated using regret (Figures (ref) and (ref)), estimator differences largely disappear. Across all outcomes—$SPV_1$, $SPV_2$, $PA\text{-}Fth_{Rep}$, and $PA_{DG}$—estimators perform nearly identically, with DM_0 achieving minimum normalized regret.
Figure (ref) reveals two distinct regret curve patterns. For $SPV_1$ and $SPV_2$, symmetric regret curves with minima near zero t-critical indicate that optimal rollout decisions require only positive effect estimates. In contrast, $PA\text{-}Fth_{Rep}$ and $PA_{DG}$ exhibit sigmoid-shaped curves: low t-critical thresholds impose minimal costs (suggesting many positive treatment effects), while high t-critical values incur substantial regret from failing to implement beneficial treatments.
\@ifstar \xxxParagraphStar \xxxParagraphNoStar *{Implications.} The dramatic divergence between MSE and regret rankings reinforces our framework's core insight: optimal estimation strategies depend on analytical objectives. For inferential goals with these outcomes, covariate adjustment with outcome-specific winsorization minimizes MSE. For rollout decisions, simple difference-in-means without winsorization minimizes opportunity costs. Organizations conducting experiments for both purposes must explicitly prioritize objectives or accept compromise solutions that perform reasonably—though not optimally—across metrics.
This paper addresses a key challenge in experimental design and causal inference: how to systematically select treatment effect estimators when faced with multiple reasonable methodological choices.
Our work makes several key contributions to the methodological literature on causal inference and experimental design. We develop a principled framework for estimator comparison that avoids the pitfalls of both ad hoc selection. We also demonstrate that optimal estimator selection depends critically on the intended use of treatment effect estimates. Through our Amazon supply chain and SDC case studies, we show that estimators optimized for inferential precision (measured by MSE) can perform poorly for decision-making applications (measured by regret), and vice versa.
Our findings juxtapose several prevailing assumptions in the experimental literature. The performance reversals we observe between MSE and regret evaluation suggest that the community's focus on p-values may not always translate to optimal decision-making performance. This disconnect is particularly relevant for applications where experiments are primarily conducted to inform operational decisions rather than to estimate parameters for scientific understanding. Our framework also highlights the importance of domain-specific methodological validation. We suggest that organizations and research communities would benefit from developing domain-specific best practices rather than relying solely on general-purpose methodological recommendations.
More broadly, our work contributes to the growing recognition that causal inference methods must be tailored to their intended applications. Just as the choice between randomized trials and observational studies depends on feasibility and ethical considerations, the choice of estimation methods within randomized trials should depend on analytical objectives and domain characteristics.
Several limitations of our current approach suggest important directions for future research. First, our framework currently focuses on treatment effect estimation for categorical treatment in cross-sectional experiment. Extensions to time varying treatment experiments, factorial designs, and more complex treatment structures would broaden the applicability of our approach. Similarly, incorporating methods for handling interference, non-compliance, and other complications common in field experiments would enhance practical relevance.
Second, while our permutation-based inference approach provides robust statistical validity, it may be computationally intensive for very large families of experiments. Developing more efficient computational approaches or approximation methods could facilitate adoption in settings with thousands of experiments.
Third, our current framework treats the choice of evaluation metric (MSE versus regret) as given, but in practice, determining the appropriate metric for a given application may itself be challenging. Future work could develop principled approaches for metric selection or multi-objective optimization approaches that balance multiple evaluation criteria.
Based on our findings, we offer several practical recommendations for researchers and practitioners conducting experimental programs:
Organizations conducting experiments primarily for decision-making should prioritize regret-based evaluation over MSE-based evaluation when selecting estimation methods. Simple methods like difference-of-means may outperform sophisticated alternatives when the goal is minimizing regreet rather than achieving precise parameter estimates.
Researchers should explicitly articulate whether their primary objective is scientific inference or practical application, as this choice should influence methodological decisions. When both objectives are important, methods like OLS that perform reasonably well across evaluation criteria may provide a sensible compromise.
The substantial performance differences we observe across evaluation criteria suggest that new methods should be evaluated using metrics aligned with their intended applications. Method developers should consider multiple evaluation criteria and acknowledge trade-offs between different objectives.
Organizations operating large-scale experimentation platforms should consider implementing our framework to develop domain-specific best practices rather than relying on one-size-fits-all recommendations. Regular validation exercises using accumulated experimental data can inform evolving methodological guidelines.
As experimental methods continue to evolve and new techniques emerge, frameworks like ours will become increasingly important for ensuring that methodological sophistication translates into improved decision-making and scientific understanding. The ultimate goal is not to find the single best method, but to develop principled approaches for selecting methods that are well-suited to specific research contexts and analytical objectives.