Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
37,262 characters · 19 sections · 40 citation commands
Anytime-Valid Inference in Adaptive Experiments: Covariate Adjustment and Balanced Power
\begingroup \footnote{Author's note: The MAD and our proposed extensions are implemented via an open-source Python package, available \hyperlink{https://pypi.org/project/pyssed/}{through PyPI} and on GitHub at \href{https://github.com/dmolitor/pyssed}{\textcolor{blue}{github.com/dmolitor/pyssed}}. Additionally, a replication package for all figures and tables is available at \href{https://github.com/dmolitor/anytime-valid-inference-in-adaptive-experiments}{\textcolor{blue}{github.com/dmolitor/anytime-valid-inference-in-adaptive-experiments}}.} \addtocounter{footnote}{-1} \endgroup
Adaptive experiments have become a key tool in modern experimental design. Adaptive experimental designs, such as multi-armed bandits (MABs), can help researchers achieve particular goals that are beyond the scope of standard randomized experiments: rapidly identifying the optimal treatment arm(s) from a larger set of treatments russo_simple_2018, gosciak_adaptive_2025, improving statistical power and inferential precision for optimal treatments offerwestort_adaptive_2021, and maximizing cumulative respondent welfare during the experiment thompson_likelihood_1933. Although adaptive experiments are far more efficient for these stated goals than randomized experiments, they introduce two notable challenges for applied researchers.
The thorniest limitation is that adaptive experiments compromise valid causal inference, making standard average treatment effect (ATE) estimators unreliable. The primary inferential challenge is that the treatment assignment $W_t$ for unit $t$ depends on prior treatment assignments and outcomes $\{(W_1, Y_1),...,(W_{t-1},Y_{t-1})\}$. This violates core assumptions that standard ATE estimators like difference-in-means and inverse propensity score estimators rely on for unbiasedness or valid inference, rendering such estimators unreliable on adaptively collected data hadad_confidence_2021, dimakopoulou_online_2021. Additionally, adaptive experiments typically concentrate sample on the optimal treatments (e.g. the treatment that maximizes respondent welfare), leaving sub-optimal treatments underpowered---a critical limitation when robust inference is needed across all treatments, not just the best-performing one.
Recent advances in adaptive experimental design address some of these challenges. One strand of work that addresses the challenge of valid inference in adaptive experiments (hadad_confidence_2021, bibaut_post-contextual-bandit_2021, zhang_statistical_2021, zhang_inference_2021) relies on reweighting schemes that restore asymptotically-valid confidence intervals. However, these methods demand that every treatment retain a positive sampling probability at each unit $t$, ruling out common adaptive designs such as Explore–Then–Commit (ETC) and Upper Confidence Bound (UCB) algorithms. Another, more recent, strand (waudby-smith_anytime-valid_2024, ham_design-based_2023) circumvents asymptotic normality altogether by employing anytime–valid inference, which maintains error control for arbitrary adaptive designs. The Mixture Adaptive Design (MAD), proposed by liang_experimental_2024, adopts this approach and resolves the challenge of valid inference in adaptive experiments by enabling unbiased estimation of the ATE and providing valid, $(1-\alpha)$ confidence sequences (CSs) for all treatment arms.\footnote{$\alpha$ is a fixed significance level, e.g. $\alpha=0.05$.} Although MAD enables valid inference in adaptive experiments, its ATE estimates often suffer from excess variance, leading to wide CSs. Additionally, MAD does not directly address the statistical power imbalance inherent in adaptive sampling, as the underlying adaptive sampling strategy still concentrates samples on optimal treatments, leaving sub-optimal treatments underpowered.
In this paper we propose modifications to the MAD estimation framework that address both of these challenges: covariate-adjusted MAD can achieve significant ATE precision improvements via covariate adjustment, and power-modified MAD adaptively re-allocates sample to under-powered treatments to ensure improved statistical power for all treatments.
It is often desirable, particularly in the context of online or sequential experiments, to continuously monitor parameter estimates in real-time with the goal of dynamically stopping data collection once a desired precision or significance criterion is met.
Confidence intervals (CIs) are well-known frequentist statistical tools that quantify the uncertainty around parameter estimates. Traditionally, a 95% confidence interval computed from a fixed sample size provides a range of plausible values for the parameter of interest, guaranteeing that, under repeated sampling, approximately 95% of such intervals will contain the true parameter. Precisely, a $(1 - \alpha)$ CI $[L, U]$ for parameter $\delta$ guarantees that for a given sample size $n$ Pr$(\delta \in [L, U]) \geq 1 - \alpha$. However, conventional CIs are only valid when $n$ is fixed beforehand; any form of continuous monitoring and dynamic stopping invalidates this inferential guarantee and inflates Type 1 error rates.
CSs are interval estimates that hold uniformly over time, i.e. anytime-valid, meaning that their Type 1 error guarantees remain valid regardless of whether, and how often, researchers inspect the data. A $(1-\alpha)$ CS is a random sequence of intervals indexed by sample size, which has at least probability $(1-\alpha)$ of containing the true parameter simultaneously over the entire sequence. Precisely, a $(1 - \alpha)$ CS $[L_n, U_n]$ for parameter $\delta$ guarantees that, Pr$(\forall n, \delta \in [L_n, U_n]) \geq 1 - \alpha$. An important implication is that CSs allow researchers to continuously monitor parameter estimates and use these estimates to dynamically stop data collection, all while maintaining Type 1 error guarantees ramdas_1_2018.
Next, we introduce a slightly weaker version of CSs---asymptotic CSs. Precisely, $C_n=[L_n, U_n]$ is a $(1-\alpha)$ asymptotic CS for parameter $\delta$ if there exists some (unknown) non-asymptotic $(1-\alpha)$ CS ($C_n^*$) for $\delta$ such that $C_n \overset{a.s.}{\to} C_n^*$. Asymptotic CSs require less-restrictive assumptions compared to their non-asymptotic counterparts and therefore facilitate applications in a wide array of settings. Although they sacrifice small-sample validity for flexibility and generalizability, asymptotic CSs have been shown to perform very well empirically, even in small- and moderate-sample settings (ham_design-based_2023; waudby-smith_time-uniform_2024; liang_experimental_2024).
In this section, we formalize our problem setting and corresponding notation.
Following the notation of liang_experimental_2024, we begin by assuming that we observe a sequence of $t$ experimental units $\{X_i,W_i,Y_i\}_{i=1}^t$ where $W_i \in \{0, ..., K-1\}$, $Y_i$, and $X_i \in \mathbb{R}^d$ are the treatment assignment ($W_i=0$ indicating assignment to the control group), observed outcome, and covariate vector for unit $i$, respectively. Although our results hold for both $K \geq 2$ (multiple treatments) as well as the batched sampling setting, for the remainder of our main discussion we will assume that we observe a single unit at a time and that we have binary treatment assignments, i.e. $W_i \in \{0, 1\}$. Throughout our analysis, we adopt the Stable Unit Treatment Value Assumption rubin_comment_1986, which assumes no population interference and that, for each unit, there is only one version of each treatment.
Given these assumptions, each unit $i$ has a pair of potential outcomes $\{Y_i(0), Y_i(1)\}$ rubin_causal_2005, and the observed outcome depends on the treatment assignment: \[Y_i=\mathds{1}\{W_i=1\}Y_i(1) + \mathds{1}\{W_i=0\}Y_i(0).\] The individual treatment effect (ITE) is defined as $\tau_i:=Y_i(1)-Y_i(0)$, and the ATE up to unit $t$ is \[\bar{\tau}_t:=\frac{1}{t}\sum_{i=1}^t{\tau_i}.\]
Following liang_experimental_2024, we adopt a design-based perspective on causal inference, as opposed to a model-based perspective. Under the design-based perspective, experimental units are drawn from a finite population which the researcher fully observes and the causal estimands of interest are specific to the actual sample gathered. In contrast, the model-based perspective assumes that experimental units are drawn from some (potentially infinite) super-population, and the estimands of interest are defined relative to this infinite super-population. Under the design-based perspective, the set of all covariate vectors and potential outcomes for the population can be treated as unknown but non-random quantities abadie_samplingbased_2020. As a result, the properties of our estimator arise solely from the randomness induced by the treatment assignment policy; that is, $\{X_i, Y_i(0), Y_i(1)\}$ is non-random, so $Y_i$ is random exclusively through the randomness induced by treatment assignment $W_i$. For a deeper examination of the design-based perspective and its application to adaptive experimental design, we refer the reader to liang_experimental_2024, Section 3.1.
We now formalize covariate-adjusted MAD (MADCovar), which extends MAD to accomodate covariate adjustment in its ATE estimates, often resulting in significant precision gains.
As is standard in the adaptive experimental literature, we assume that treatment assignment for a given unit $t$ can depend on previously observed outcomes and treatment assignments up to $t-1$. Formally, define the history at unit $t$ as $\mathcal{H}_t := \{Y_i, W_i, X_i\}_{i=1}^t$, where $Y_i, W_i, X_i$ are previously observed outcomes, treatment assignments, and covariate vectors, and let $\mathcal{F}_t$ be the sigma-algebra containing all pairs of potential outcomes $\{Y_i(1), Y_i(0)\}_{i=1}^t$ as well as $\mathcal{H}_{t-1}$. Additionally, let $\mathcal{A}$ be an adaptive treatment assignment policy, such as Thompson sampling thompson_likelihood_1933 or the Upper Confidence Bound algorithm, and define the probability that $\mathcal{A}$ assigns unit $t$ to treatment $w$ as $p_t^\mathcal{A}(w) := \mathbb{P}_\mathcal{A}(W_t=w|\mathcal{H}_{t-1})$. Finally, let $\delta_t \in (0, 1]$ be a deterministic sequence which, for each unit $t$, determines the tradeoff between a fully random assignment policy and $\mathcal{A}$, with the requirement that $\delta_t = \omega\left(\frac{1}{t^{1/4}}\right)$.\footnote{By definition, $\delta_t = \omega\left(\frac{1}{t^{1/4}}\right)$ implies that $\frac{1}{\delta_t} = o\left(t^{1/4}\right)$. Intuitively, $\delta_t$ is a function of $t$ that converges to 0 slower than $\frac{1}{t^{1/4}}$.}
A natural extension of Definition (ref) to the $K \geq 2$ treatment setting with $w \in \{0, 1, ..., K-1\}$ is to define $p_t^\text{MAD}(w) := \mathbb{P}_\text{MAD}(W_t=w|\mathcal{H}_{t-1})=\delta_t\left(\frac{1}{K}\right) + (1-\delta_t)p_t^\mathcal{A}(w)$. Definition (ref) can also be easily adapted for batched assignment, for which we refer the reader to Definition 5 (Section 4.1) of liang_experimental_2024.
We now define our ATE estimator. We adopt the following covariate-adjusted estimator of $\tau_i$ in the manner of robins_semiparametric_1995:
where $\hat{\mu}_w(X_i)$ is an arbitrary estimator of $\mathrm{E}[Y_i(w)|X=X_i,\mathcal{F}_{i-1}]$. Under the design-based perspective, it follows that $\mathrm{E}[\hat{\tau}_i|\mathcal{F}_i]=\tau_i$, where the expectation is taken with respect to the treatment assignment mechanism, and thus $\hat{\bar{\tau}}_t:=\frac{1}{t}\sum_{i=1}^t{\hat{\tau}_i}$ is an unbiased estimator for $\bar{\tau}_t$. Additionally, we have that, for all $i$, \[\mathrm{Var}[\hat{\tau}_i|\mathcal{F}_i] \leq \sigma^2_i, \text{ where } \sigma^2_i:=\frac{(Y_i(1)-\hat{\mu}_1(X_i))^2}{p_i^\text{MAD}(1)}+\frac{(Y_i(0)-\hat{\mu}_0(X_i))^2}{p_i^\text{MAD}(0)},\] and thus, a natural unbiased estimator for $\sigma^2_i$ is \[\hat{\sigma}^2_i := \frac{\mathds{1}\{W_i=1\}(Y_i - \hat{\mu}_1(X_i))^2}{(p_i^\text{MAD}(1))^2} + \frac{\mathds{1}\{W_i=0\}(Y_i - \hat{\mu}_0(X_i))^2}{(p_i^\text{MAD}(0))^2}.\] For future definitional purposes, we will define $S_t := \sum_{i=1}^t\sigma_i^2$ and $\hat{S}_t := \sum_{i=1}^t\hat{\sigma}_i^2$.
We now state the necessary assumptions to establish the primary inferential results.
Note that this assumption applies to the fitted values of the outcome models $\hat{\mu}_w$ evaluated on realized covariate vectors. In practice, this assumption is almost always satisfied: any reasonable outcome model will produce bounded predictions. In the rare event that fitted values exceed floating-point limits, researchers can safely cap them without losing meaningful information. Notably, our CSs do not depend on knowing $N$, and its value does not affect CS width.
Alongside Assumption (ref), we also adopt the Bounded Realized Potential Outcomes and At Least Linear Growth of Cumulative Conditional Variance assumptions (Assumptions 1 and 2) of liang_experimental_2024. We now state the primary result:
We defer the proof to Appendix (ref). At a high level, the proof of Theorem (ref) follows by showing that $\delta_t = \omega(t^{-1/4})$---together with Assumptions 1 and 2 of liang_experimental_2024 and our Assumption (ref)---ensures that $\hat{\bar{\tau}}_t$ satisfies a Lindeberg-type uniform integrability condition (see Lemma A.1 of liang_experimental_2024). This allows us to apply the universal asymptotic CS of waudby-smith_time-uniform_2024.
In practice, Theorem (ref) lets researchers use MAD to combine the efficiency gains of adaptive experiments with valid, sequential ATE inference, while boosting inferential precision through covariate adjustment via flexible outcome models $\hat{\mu}_w(x)$.
Finally, we will briefly touch on a number of relevant properties of MAD that hold for all our presented results. Since these properties are derived directly from MAD, we will in all cases refer the reader to the relevant discussion in liang_experimental_2024. \paragraph{Non-stationarity} Theorem 2 of liang_experimental_2024 guarantees that, under the conditions of Theorem (ref), if $\bar{\tau}_t \to c$ for some $|c| > 0$ as $t \to \infty$, the sequence $(\hat{\bar{\tau}}_t \pm \hat{V}_t)$ will exclude 0 in finite time with probability 1. Intuitively, our CSs will allow the researcher to correctly reject the point null, $\mathcal{H}_0:\bar{\tau}_t=0$, in finite time. This property allows for non-stationarity as long as, in the long run, $\bar{\tau}_t$ converges to some value. For a thorough discussion of the implications of non-stationarity, particularly as it pertains to various stopping rules, see Section 5.1 of liang_experimental_2024. \paragraph{Inference for multiple treatments and batched assignment settings} As noted previously, all prior theoretical results extend directly to settings with $K \geq 2$ (multiple treatments) and to settings with batched assignment (Definition 5, Liang and Bojinov). To see a detailed formulation of these extensions, see Appendix (ref). The extension to settings with many treatments is crucial since, in practice, adaptive experiments are often deployed in settings with multiple treatments. As a result, we expect that most practitioners will be applying these methods in settings with $K \geq 2$. \paragraph{Setting $\delta_t$} The deterministic sequence $\delta_t$ plays a critical role in the experimental design. As noted previously, $\delta_t$ determines the tradeoff between a fully random assignment policy and the adaptive assignment policy $\mathcal{A}$. As a result, liang_experimental_2024 note that the regret of MAD (and thus our extensions) at any unit $t$ is a weighted sum of the regret of a fully random design and the regret of $\mathcal{A}$, where the weights are determined by $\delta_t$. Section 5 of liang_experimental_2024 provides a detailed, finite-sample characterization of the regret of MAD, and Section 6 provides a robust set of recommendations for choosing $\delta_t$ to achieve various exploration/exploitation goals.
Using simulated data as well as data from a recent, large-scale randomized control trial (RCT), we demonstrate that MADCovar can achieve considerable precision gains relative to MAD. Additionally, these precision gains are robust to including (potentially many) irrelevant covariates in the outcome models.
Using a grid of simulations, we evaluate MADCovar’s performance across increasing levels of covariate signal strength and number of irrelevant covariates included in our outcome models. This is motivated by the empirical concern that outcome–model–based estimators can suffer variance inflation when provided with many unrelated covariates.
We simulate a simple setting with a single treatment and control group; $w \in \{0, 1\}$. Potential outcomes are defined as \[
\] where \[ X_{1,i},X_{2,i},X_{3,i} \sim \mathcal{N}(0,1) \text{ i.i.d.}, \quad \varepsilon_i \sim \mathcal{N}(0,1), \] and $\gamma \in [0, 1]$ controls covariate signal strength. We vary \(\gamma\) at three levels---low (\(\gamma=0.1\)), medium (\(\gamma=0.5\)), and high (\(\gamma=1.0\))---to assess precision gains at various levels of covariate signal strength. We add irrelevant covariates \(X_{\text{irr},i}\sim\mathcal{N}(0,1)\) in three levels:
We set Thompson sampling as the underlying adaptive assignment algorithm (\(\mathcal{A}\)), a quickly-decaying sequence \(\delta_t = \frac{1}{t^{1/5}}\), and OLS for the outcome models (\(\hat{\mu}_w\)) fit on the full set of covariates $\{X_{1, i}, X_{2, i}, X_{3, i}, X_{\text{irr},i}\}$. Finally, we simulate MAD and MADCovar for 10,000 units and calculate the corresponding \(\hat{\bar{\tau}}_t\) estimates and 95% CSs (see Figure (ref)). We repeat this process for 100 simulations and compare the mean CS interval width across these simulations for MAD and MADCovar (see Figure (ref)).
In settings with medium and high covariate signal, MADCovar provides precision gains between 40-70% on average. Additionally, these gains remain consistent even when including many irrelevant covariates in the outcome model estimation. When utilizing many covariates (particularly unrelated covariates) in very small sample sizes ($N\leq1,000$), MADCovar may suffer from worse precision than MAD, however this is quickly rectified as the sample grows relative to the covariate space. This challenge can also be rectified by choosing appropriate outcome models, such as regularized regression models.
To further illustrate the substantial precision gains of MADCovar, we move beyond simulations and leverage large-scale RCT data from a mega-study by voelkel_megastudy_2024. This study investigates the causal effects of various informational interventions on feelings of partisan animosity among Democrats and Republicans. The study evaluates roughly 25 interventions using a fully randomized design, assigning approximately 1,000 respondents to each treatment and approximately 5,000 to the control group. The respondents are asked to record their feelings of partisan animosity on a numeric scale from 1-100 (higher meaning more animosity) both before and after the intervention. The study also collects a rich set of respondent characteristics that are posited to be predictive of the individual's level of partisan animosity. Using this dataset, we simulate an experiment by applying MAD and MADCovar to a random subset of 13 interventions, sampling units with replacement, to match the original RCT’s total sample size of $\approx 32,000$. As before, we set \(\mathcal{A}\) as Thompson sampling, \(\delta_t = \frac{1}{t^{1/10}}\), and OLS for the outcome models fit on the full set of respondent characteristics. At the conclusion of the experiment, we compare ATE estimates and 95% CSs of MAD and MADCovar relative to the ground truth (RCT estimates). Figure (ref) demonstrates that the precision advantages of MADCovar extend to real-world scenarios, and are equally as striking as the precision gains observed in the simulations. In this example, covariate-adjusted ATE estimation allows us to achieve up to 60-75% precision gains relative to the non-covariate-adjusted baseline.
Next, we introduce Power-modified MAD (MADMod) to address the challenge of imbalanced statistical power that is inherent in adaptive experiments. MADMod incorporates dynamic sample allocation to underpowered treatments, ensuring improved statistical power across all treatments. This allows researchers to balance the efficiency gains of adaptive experiments with the need for robust inference, enabling researchers to conduct experiments that are both statistically robust and resource efficient.
Let $K$ denote the set of treatment arms. For each arm $k \in K$ and unit $t$, we introduce importance weights $w_{k}(t) \in [0, 1]$ which are a function of $t$. Once the estimated ATE for arm $k$ reaches statistical significance, or equivalently the CS for arm $k$ excludes 0, $w_{k}(t)$ begins to shrink at a researcher-defined rate. By construction, the probabilities $\{p_t^\mathcal{A}(k)\}_{k \in K}$ form a valid probability distribution: $\sum_{k \in K}{p_t^\mathcal{A}(k)}=1$. We modify this distribution using a re-weighting function $g$, which adjusts $p_t^\mathcal{A}(k)$ based on its importance weight $w_{k}(t)$.
The re-weighted probability $g(p_t^\mathcal{A}(k)) := p_t^g(k)$ for each treatment arm $k$ at time $t$ is then computed as follows:
This transformation preserves the probability distribution $\big(\sum_{k \in K}{p_t^g(k)}=1\big)$ while allowing controlled reallocation of samples toward underpowered arms.
While researchers can specify $w_k(t)$ as any non-increasing function of $t$, this function provides a flexible way to balance between two extremes: fully adhering to the underlying adaptive algorithm’s ($\mathcal{A}$) assignment probabilities or completely redirecting probability mass away from statistically significant arms. Setting $w_{k}(t) = 1$ retains the original adaptive probabilities, while $w_{k}(t) = 0$ eliminates allocation to statistically significant arms entirely. We discuss design choices that strike a middle ground between these two.
For researchers seeking modest power improvements while preserving the adaptive algorithm’s efficiency (i.e., favoring exploitation over exploration), a slowly decaying function such as $w_k(t) = \frac{1}{t^{1/10}}$ may be suitable. In contrast, if the primary goal is to ensure that as many treatments as possible are well-powered, a more rapidly decaying function like $w_k(t) = \frac{1}{t}$ is more appropriate. Alternatively, if the goal is to apply a fixed discount (e.g., 30%) to the importance weights of statistically significant treatments, a function such as $w_k(t)=0.7$ achieves that. In general, users can select from a variety of decay functions to tailor this tradeoff to their specific goals.
Using simulations with $K=4$ treatment arms, we demonstrate MADMod’s ability to achieve precise, anytime-valid inference while improving statistical power across all treatments. Potential outcomes for the control arm are generated as i.i.d. draws from $Y_t(0) \sim \text{Bernoulli}(0.5)$, while treatment potential outcomes follow i.i.d. draws from $Y_t(k) \sim \text{Bernoulli}(p_k)$ for $k \in {1,2,3,4}$. We set $p=(0.6, 0.62, 0.8, 0.82)$ to create a clear separation between suboptimal arms (1 and 2) and near-optimal/optimal arms (3 and 4), corresponding to ATEs of 0.1, 0.12, 0.3, and 0.32. We set Thompson sampling as $\mathcal{A}$, and define $\delta_t = \frac{1}{t^{0.24}}$. Finally, we define a slowly-decaying importance weight function, $w_k(t)=\frac{1}{t^{1/8}}$, to achieve improved statistical power while still prioritizing the efficiency goals of the underlying Thompson sampling algorithm. We run 100 paired simulations comparing MAD and MADMod over 20,000 units with early stopping when all ATEs are detected as statistically significant.
Figure (ref) shows that MADMod consistently improves Type 2 error control and precision across all treatment arms, with particularly large gains in error control for suboptimal treatments (treatment arms 1 and 2). Taken together, decreased Type 2 error and narrower interval widths allows for well powered and precise inference on all arms.
As Figure (ref) shows, these gains come with a tradeoff in sample allocation. MADMod assigns significantly more samples to suboptimal arms than MAD, sacrificing some of the efficiency of $\mathcal{A}$ to ensure reliable inference for all arms. This tradeoff is often desirable, as researchers typically seek well-powered estimates for all treatment effects, not just the optimal arm.
Our work offers methodological improvements that make anytime-valid inference in adaptive experiments more effective in applied settings. By increasing precision without requiring larger sample sizes, covariate-adjusted ATE estimation makes experiments more efficient and ethical, reducing costs, accelerating discovery, and limiting participant exposure to inferior treatments. These benefits are particularly important in high-stakes settings such as healthcare, education, and social policy. Moreover, improving statistical power across all treatments enables researchers to assess the effectiveness of every intervention---not just the most promising ones---reducing the risk of discarding treatments that, while not optimal, may still be valuable in practice.
Despite these clear benefits, our methods also face some limitations.
Anytime-valid inference is a powerful feature, enabling researchers to continuously monitor ATE estimates and stop data collection once a desired level of precision or statistical significance is reached. However, this flexibility comes with a trade-off in precision: for any fixed sample size $N$, the confidence sequence $\text{CS}_N$ will be wider, and thus less precise, than a corresponding confidence interval $\text{CI}_N$. In some cases, researchers may pre-specify $N$ in a non-data-driven way, whether due to ethical or fairness considerations, power calculations, or logistical constraints. In such settings, our CSs may offer less value, as they sacrifice precision for the anytime-valid property. Researchers may instead prefer fixed-sample ATE estimators that yield asymptotically valid CIs under adaptive sampling (e.g. hadad_confidence_2021), offering greater precision when the stopping rule is fixed in advance.
simchi-levi_multi-armed_2024 present a closely related approach that offers a rigorous theoretical treatment of the efficiency/power trade-off in adaptive experiments. They frame the problem as a minimax multi-objective optimization task and derive a Pareto-optimal solution, where improving either efficiency or statistical power necessarily compromises the other. In contrast, MADMod addresses the same trade-off through an algorithmic lens, allowing researchers to specify a function that governs how rapidly they trade efficiency for power.
While this flexibility comes at the cost of theoretical optimality, it offers important advantages. The method proposed by simchi-levi_multi-armed_2024, though grounded in theory, is not broadly applicable across adaptive designs. Their solution is implemented via a new MAB algorithm tailored to their specific problem setting, which may not align with a researcher’s goals. MADMod, by contrast, inherits the versatility of MAD and can be integrated with virtually any adaptive experimental design. This allows researchers to retain control over the experimental strategy while still reaping the benefits of principled power reallocation.
In this paper, we advance methods for anytime-valid causal inference on the ATE in adaptive experiments, addressing key challenges for applied researchers. First, we introduce MADCovar, which leverages covariate adjustment to substantially improve ATE precision. Second, we present MADMod, which adaptively reallocates sample to under-powered treatment arms, enhancing statistical power across all arms. Together, these methods give practitioners greater flexibility and control over both inferential precision and statistical power in adaptive experimental settings.