Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
71,684 characters · 7 sections · 54 citation commands
Testing Effect Homogeneity and Confounding in High-Dimensional Experimental and Observational Studies
\singlespacing
Experiments are widely considered the benchmark for credible causal inference because randomization eliminates confounding and delivers unbiased estimates of treatment effects. In practice, however, experimental data often come with important limitations: sample sizes tend to be modest, covariate information is restricted, and external validity is frequently uncertain. By contrast, modern observational data sets are typically much larger and richer, offering detailed high-dimensional covariates and broad population coverage. These features make them attractive for studying treatment effect heterogeneity, but causal interpretation is hindered by the possibility of unobserved confounding. The increasing availability of both experimental and observational data raises an interesting question: to what extent do treatment effect estimates align across data sources once we condition on observed covariates, and what does this reveal about the internal and external validity of identification strategies in experiments and observational studies?
This paper proposes a framework for testing the homogeneity of conditional average treatment effects (CATEs), given covariates, across multiple experimental and observational studies (sites) such as different regions or countries. When applied solely to randomized experiments, where unobserved confounding is ruled out by design, our test allows assessing treatment effect heterogeneity arising from unobservables that differ across experiments. If CATEs are homogeneous across experiments, this indicates an absence of interactions between the treatment and unobserved factors in the average treatment effect, implying that CATEs are externally valid across experimental settings. Moving beyond experiments, comparing CATEs between experimental and observational data additionally permits assessing the presence of confounding. If the CATE estimates are asymptotically equivalent across experiments and observational studies, then CATEs are unconfounded by unobservables (i.e., internally valid) and homogeneous across settings (i.e., externally valid). Conversely, if they differ, the discrepancy may stem from unobserved confounding, effect heterogeneity, or both. This may motivate a sequential application of the testing approach; first, across experiments to assess effect homogeneity, and, if homogeneity is not rejected, subsequently across experimental and observational studies to assess unconfoundedness in addition to external validity.
Our testing approach builds on the double machine learning (DML) framework Chetal2018, which combines doubly robust (DR) treatment effect estimation Robins+94, RoRo95, Ha98 with machine learning to flexibly adjust for high-dimensional covariates. More specifically, we extend the Neyman1959-orthogonal score function introduced by apfel2024learningcontrolvariablesinstruments, who propose a test for whether CATEs within a single study are jointly zero. In contrast, we use an analogous orthogonal formulation to test whether differences in CATEs across studies, experimental and/or observational, are jointly zero. The resulting test is $\sqrt{n}$-consistent (where $n$ denotes the sample size) and asymptotically normal under specific regularity conditions, in particular when the machine learning estimators for the treatment and outcome models converge at rate $o(n^{-1/4})$.
As an additional methodological contribution, we extend the framework to settings where identification relies on alternative strategies. These include instrumental-variable designs that identify the local average treatment effect (LATE) for compliers whose treatment status responds to the instrument Imbens+94, Angrist+96, as well as panel-data settings that rely on parallel trends to identify the average treatment effect on the treated (ATET) based on difference-in-differences Snow1855. Because both the conditional LATE and the conditional ATET pertain to specific subpopulations, rather than the full population conditional on covariates, testing homogeneity is informative for evaluating whether these local effects can be extrapolated to the total population, in the spirit of AnFe2010 and AronowCarnegie2013.
We then investigate the finite-sample behavior of our testing approach in a simulation study that mimics settings with multiple experimental and/or observational sites. First, we study the finite-sample size and power of the test in a multi-site experimental design by comparing CATEs across randomized sites under data-generating processes with homogeneous versus heterogeneous CATEs. Second, we consider a mixed design with randomized treatment in some sites and observational identification in others; we introduce unobserved confounding in the observational sites and evaluate the ability of the test to detect cross-site CATE differences both in the absence and presence of confounding. For each design, we run $1000$ Monte Carlo replications and report summary statistics that describe the performance of our method, such as the rejection rate, standard deviation, and sample size.
Furthermore, we provide an empirical application for our testing approach using data from the International Stroke Trial, a large multi-country randomized controlled trial which evaluated whether early aspirin allocation to patients with acute ischaemic stroke improved patient health after they had experienced a stoke. We treat countries as separate experimental sites and test whether the conditional effect of randomized aspirin assignment on six-month death or dependency is homogeneous across countries after adjusting for baseline patient characteristics.
A growing literature combines experimental and observational data to improve causal inference colnet2024causal. Existing studies use such designs to (i) generalize randomized trial findings to broader populations cole2010generalizing, Pearl_Bareinboim_2011, ghassami2022combining, stuart2015assessing, pmlr-v180-hatt22a, van2023estimating, park2024informativeness, NBERw33817, imbens2025long, parikh2025doubleML, triantafillou2023learning, lelova2025testing, (ii) increase statistical efficiency, especially for heterogeneous treatment effects yang2020improved, wu2022integrative, yang2023elastic, cheng2021adaptive, yang2025cross, rosenman2022propensity, rosenman2023combining, hatt2022combining, Brantner_2024_multipleRCT, epanomeritakis2025choosing, and (iii) diagnose or correct bias in observational analyses by using experimental benchmarks kallus2018removing, yang2020improved, wu2022integrative, yang2023elastic, liu2025causal, parikh2025doubleML, triantafillou2023learning, lelova2025testing, chen2025causal.
We complement these efforts by introducing a framework that tests whether CATEs are homogeneous across multiple experimental and observational sites. By comparing CATEs across experiments, the test can detect heterogeneity across experimental sites driven by unobservables and evaluate the external validity of experimental CATEs. While by comparing CATEs from experiments and observational sites, our test can diagnose the presence of hidden confounding and assess internal validity of treatment effects. Therefore, our approach provides a unified way to assess both internal and external validity of CATEs using distinct research designs.
The remainder of this paper is organized as follows. Section (ref) provides a detailed literature survey on studies combining multiple experimental and observational data for treatment effect evaluation. Section (ref) introduces the identifying assumptions and outlines our method. Section (ref) extends the testing framework to instrumental and panel data contexts. Section (ref) provides a simulation study that investigates the finite sample performance of our proposed test. Section (ref) illustrates the method using the International Stroke Trial. Section (ref) concludes.
Our study contributes to the three strands of research mentioned above by combining experimental and observational evidence to address unobserved confounding and to evaluate the internal as well as the external validity of CATEs. Within this body of work, the literature largely follows two approaches: (i) one explicitly models and estimates a confounding function by pooling information from RCTs and observational datasets, (ii) the other develops methods that combine or pool CATE estimators from experimental and observational samples using adaptive weights chosen to balance bias and efficiency.
The first approach focuses on the confounding function, defined as the conditional gap between causal and observational treatment effects, as a central object of interest. Early contributions such as kallus2018removing propose a method that first learns an observational CATE function and then estimates a low-dimensional correction term using experimental data, so that the observational estimate matches the randomized benchmark even with only partial covariate overlap. Building on this idea, yang2020improved introduce a data fusion framework in which both the CATE and the confounding function are identifiable once observational and experimental samples are coupled. They show that their method improves efficiency relative to cases where there are only experimental samples. Subsequent work by wu2022integrative extend this framework by proposing an R-learner that incorporates flexible machine learning methods to approximate the CATE, the confounding function, and other nuisance components.
Even more recently, yang2023elastic introduce a test-based elastic approach for integrating trial and real-world data. Their method first uses the RCT as a benchmark to test whether the observational sample suffers from bias. If the test fails, only the data from the RCT is used, but if the test supports comparability, the two sources are combined for efficiency. Complementing these elastic integration ideas, liu2025causal develop a direct hypothesis test for unconfoundedness by comparing treatment–outcome contrasts estimated from the RCT and the observational data. Unlike yang2023elastic, which couples a pretest with an adaptive estimator, liu2025causal focuses on diagnosis by flagging when the observational sample is likely confounded before applying fusion or machine learning estimators that assume ignorability. parikh2025doubleML push this diagnostic idea further by asking which assumption breaks when the two sources disagree: they develop a double machine learning framework by introducing a statistical quantity that distinguishes failures of ignorability in the observational sample from failures of external validity of the experiment.
A complementary Bayesian approach incorporates uncertainty about when observational data are safe to use. triantafillou2023learning propose Bayesian CATE estimation that adaptively borrows from observational data, while lelova2025testing study identification and transportability of CATEs under an unknown causal graph when combining experimental and observational samples. Beyond the econometric literature, related work in computer science by hatt2022combining proposes a representation learning framework which first learns the shared covariate structure from observational data and then uses experimental data to calibrate the estimation of treatment effects. They formalize the bias from unmeasured confounding as a confounding function, learn this bias by comparing observational and experimental predictions, and then use it to debias CATE estimates.
The second strand of research avoids modeling a confounding function and instead combines CATE estimators computed separately in experimental and observational samples. cheng2021adaptive combine kernel-based CATE estimates using data-driven weights that default to the experimental data when bias is suspected and combine both experimental and observational sources when estimates align. yang2025cross generalize this idea by choosing the weights given to each data source through cross-validation in a joint loss framework, trading off bias and variance across the two sources.
Related work by rosenman2022propensity propose to combine RCTs and observational data by stratifying on the observational propensity score and placing experimental units into the same strata. Within each stratum, they estimate treatment effects from experimental and observational sources and then merge them either by spiking experimental data into observational bins or through a data-driven weighting scheme that balances bias and variance. In subsequent work, rosenman2023combining extends this approach using Stein-type shrinkage estimators. Their method adaptively shrinks observational estimates toward unbiased experimental estimates. They show that these estimators reduce the mean squared error relative to performing only experimental analyzes. While this line of work focuses on optimally combining experimental and observational evidence to improve efficiency, it implicitly assumes that treatment effects are sufficiently stable across settings.
In contrast, chen2025causal investigates whether causal machine learning methods can produce reliable CATE estimates using data from two large RCTs. They show that individualized treatment effects derived from a wide range of machine learning methods fail to replicate across training and test splits or across trials, even in the absence of confounding. This highlights the difficulty of obtaining externally valid CATE estimates and the need for systematic approaches to test the stability of CATEs across settings.
For a broader synthesis of the literature, colnet2024causal provides a systematic review of approaches that integrate experimental and observational data. Additionally, Brantner2023Integration provide a review of methods to combine multiple RCTs or RCTs with observational data, with a focus on treatment effect heterogeneity. They classify approaches by the type of data available and discuss both parametric and machine learning strategies for estimating CATEs. A key takeaway is that comparing CATEs across sources provides a way to assess stability and detect potential confounding.
Despite advances in combining experimental and observational data, most applications focus on a single RCT paired with one observational dataset. As a result, little is known about whether CATEs align across multiple experiments and observational sources. An exception is Brantner_2024_multipleRCT, develop and who study methods for estimating CATEs when several RCTs are available. They adapt S-learner, X-learner, and causal forest estimators to the multi-trial setting. They show that strategies allowing trial-level heterogeneity outperform naive pooling that ignores study differences. Their work highlights the challenges of integrating multiple experiments, but does not examine the alignment of CATEs between experimental and observational data. Our study fills this gap by focusing on testing the homogeneity of CATEs across both experimental and observational sources.
$D$ denotes the binary treatment and $Y$ the outcome of interest. Using the potential outcomes framework as advocated in Neyman23 and Rubin74, we denote by $Y(d)$ the potential outcome when exogenously setting the treatment $D$ of a subject to value $d \in {1,0}$. More generally, we will use capital letters for random variables and lower case letters for their realizations. By representing the potential outcome $Y(d)$ as a function solely of a subject’s own treatment status $D=d$, we implicitly assume that the potential outcomes of one subject are not influenced by the treatment status of others. This assumption is known as the stable unit treatment value assumption (SUTVA), see Rubin80 and Cox58, and is invoked throughout. Furthermore, let $X$ denote a set of observed pretreatment covariates, and let $Z$ be a discrete variable that indexes different setups or studies in which the experimental or observational data were collected (for example, sites or regions). The variable can take integer values $z \in {1,...,L}$, with $L$ denoting the number of setups.
We suggest a method to test effect homogeneity in conditional average treatment effects (CATEs) across different experiments, or selection-on-observables (and effect homogeneity) across experimental and observational data, respectively. First, we consider the case of comparisons within experimental studies. Suppose that treatment is randomly assigned within each site or region $Z$, possibly conditional on covariates $X$ (as in stratified randomization). This corresponds to the standard selection-on-observables assumption, also known as unconfoundedness or conditional independence Im04.
where ${\perp\!\!\!\perp}$ denotes statistical independence.
In addition, we require a condition ensuring that treated and untreated units are observed in all relevant subpopulations of $X$ and $Z$:
The common support assumption guarantees overlap in the treatment assignment across different experiments and covariate profiles. It rules out situations where, conditional on covariates $X$, treatment assignment or assignment to a specific experiment is deterministic.
The conditional average treatment effect (CATE) given covariates $X$ and experiment $Z$ is defined as
Under (conditional) treatment randomization, which implies the satisfaction of Assumption (ref), we have that $\Delta_{X,Z}$ corresponds to
We are interested in testing whether these conditional effects vary across experiments. Formally, we impose:
This assumption states that CATEs are homogeneous across experiments $Z$. Such homogeneity may hold for two distinct reasons. First, treatment effects may not interact with unobserved heterogeneity once we condition on $X$. An example where this is satisfied is the following structural model:
where $U$ denotes unobserved characteristics and $\kappa$ and $\eta$ are unknown functions. While the effect of $D$ may vary arbitrarily across $X$, it does not vary with $U$ conditional on $X$ due to the additive separability of $\kappa$ and $\eta$. In such cases, even if unobserved characteristics differ across experiments, they do not generate treatment effect heterogeneity.
Second, treatment effects may indeed interact with unobserved heterogeneity, but the distribution of this heterogeneity remains stable across experiments. For instance, consider the structural model
where the effect of $D$ on $Y$ may arbitrarily interact with $U$. This second case, effect heterogeneity existing but not being detectable because the distribution of $U$ is identical across experiments, appears less plausible when experimental sites differ substantially in institutional, geographic, or temporal contexts. Such differences typically shift the distribution of unobserved characteristics. The more variation there is in unobserved heterogeneity across experiments, the more informative the data become for detecting interactions between treatment effects and unobservables.
Conditional on Assumption (ref), Assumption (ref) yields the following testable null hypothesis:
where $\mathcal{X}$ denotes the support of $X$ and $\mu_{d,x,z}=E[Y|D=d,X=x,Z=z]$ denotes the conditional mean outcome.
To test the null hypothesis in (ref), we adapt the doubly robust conditional independence test of apfel2024learningcontrolvariablesinstruments to the problem of effect homogeneity. That is, we extend their approach, which tests whether CATEs differ from zero, to instead test whether differences in CATEs across experiments are equal to zero. Note that (ref) can equivalently be written as
where $z^-$ denotes values of $Z$ different from $z$, such that $Z \neq z$.
Denote by $p_{d,z}(X)=\Pr(D=d,Z=z|X)$ the joint propensity of treatment and being in a specific experiment or site. We denote the nuisance parameters by
$\eta=(p_{1,z}(X),p_{1,z^-}(X),\mu_{1,X,z},\mu_{1,X,z^-},p_{0,z}(X),p_{0,z^-}(X),\mu_{0,X,z},\mu_{0,X,z^-})$.
Modifying apfel2024learningcontrolvariablesinstruments, who consider simple difference in conditional means to test whether any CATE is different from zero, to double (or differences in) differences, to test whether any difference in CATEs (across experiments) is different from zero. More concisely, testing with a multivalued $Z$ can be based on the following score function, in which $O=(Y,D,X,Z)$ denotes the random variables:
This score has a variance that is bounded away from zero, is zero in expectation under the null hypothesis in equation (ref) when $\theta_0=0$, and is Neyman-orthogonal, see proof provided in Appendix of apfel2024learningcontrolvariablesinstruments. This follows directly from the proofs in apfel2024learningcontrolvariablesinstruments, as our score function is based on applying their type of score function twice to turn it into a double (rather than a single) difference across $\mu_{d,X,z}$. As a the double difference is just a linear combination of the single differences, the asymptotic findings in apfel2024learningcontrolvariablesinstruments directly apply to our case, too. In particular, cross-fitted estimators of $\theta_0$ based on the score function (ref) is asymptotically normal and $\sqrt{n}-$ consistent under specific regularity conditions, in particular if machine learners used for estimating nuisance parameters $\eta$ have a convergence rate of $o(n^{-1/4})$.
The same testing framework can also be applied for comparing experimental and observational studies in a second step following the within-experiments comparison. In purely observational data, where Assumption (ref) cannot be taken for granted, rejection of (ref) or (ref) may reflect violations of Assumption (ref), Assumption (ref), or both. However, if experimental data suggest that CATEs are homogeneous, i.e., Assumption (ref) holds, then comparisons of CATEs between experimental and observational studies provide a direct test of Assumption (ref). Specifically, one can define $Z$ such that $Z=1$ indicates observations from experimental studies, while $Z=2,\dots,L$ index observational studies. Alternatively, $Z$ may distinguish between observational studies only. In both cases, testing can again be implemented using the score in (ref).
Our method for testing effect homogeneity can also be adapted to settings where treatment is not conditionally exogenous. For instance, access to treatment may be randomly assigned conditional on $X$, while actual treatment take-up deviates from assignment due to noncompliance. In this case, we may use assignment, henceforth denoted by $W$, as an instrument for actual treatment $D$. Following Imbens+94 and Angrist+96, we denote by $D(w)$ the potential treatment as a function of instrument $W$ and by $Y(w,d)$ the potential outcome as a function of $W$ and $D$. We impose the following instrumental variable (IV) assumptions conditional on covariates $X$ and experiment $Z$, see Abadie00:
The first line of Assumption (ref) requires that the instrument is as good as randomly assigned and satisfies the exclusion restriction conditional on $X$ and $Z$. The second line rules out the existence of defiers, but it also requires the existence of compliers conditional on $X$, due to the nonzero conditional first stage. The third line imposes common support on the instrument, implying that assignment is not deterministic in $X$ and $Z$.
Assumption (ref) permits identifying conditional local average treatment (CLATE) effect among the subgroup compliers, denoted by $c$, who are treated only if the instrument is equal to one: $c: D(1)=1,D(0)=0$. The CLATE given covariates $X$ and experiment $Z$ is defined as
The CLATE is identified using a Wald-type estimand Wald40, defined as the ratio of the reduced-form effect of the instrument on the outcome to the first-stage effect of the instrument on the treatment, conditional on $X$ and $Z$:
where we define the reduced-form and first-stage effects as $\bar g_{x,z} = m_{1,x,z} - m_{0,x,z}$ and $\bar h_{x,z} = r_{1,x,z} - r_{0,x,z}$, with $m_{w,x,z} = E[Y\mid W=w,X=x,Z=z]$ and $ r_{w,x,z} = E[D\mid W=w,X=x,Z=z]$.
Considering, in analogy to equation (ref) for the CATE, the following null hypothesis,
permits testing the following effect homogeneity assumption among compliers:
Since $\delta_{c,x,z} = g_{x,z}/h_{x,z}$, we note that the null hypothesis (ref) can be equivalently written in cross-multiplied form as
Hence, a ratio-free version of the null hypothesis for the CLATE that is analogous to equation (ref) for the CATE is given by
where $z^-$ denotes values of $Z$ different from $z$. We also define the propensity score $\pi_{w,z}(x) = \Pr(W=w,Z=z\mid X=x)$ and collect the nuisance parameters in $\eta = (m_{w,x,z},r_{w,x,z},\pi_{w,z}(x),\pi_{w,z^-}(x))_{w\in\{0,1\},z\in\{1,\dots,L\}}$.
Following a similar logic as in the CATE case, we construct a DR score function based on (ref) for the CLATE setting. To this end, define the DR augmentations for the reduced form and first stage effects as
Using these, we denote the cross-product difference by
Denoting by $O=(Y,D,X,Z,W)$ the random variables, a DR score function for testing the null hypothesis in (ref) is
This score function has zero mean under the null hypothesis $H_0:\theta_0=0$ and is Neyman-orthogonal with respect to the nuisance parameters $\eta$, see the proof provided in Appendix (ref). Consequently, cross-fitted estimators of $\theta_0$ based on (ref) are $\sqrt{n}$-consistent and asymptotically normal under specific regularity conditions, in particular if machine learning estimators of the nuisance functions converge at rate $o(n^{-1/4})$.
We note that, although the construction of the CLATE score function is related to the approach of apfel2024learningcontrolvariablesinstruments, there is a conceptual difference compared to the CATE framework considered in their paper and in our Section (ref). While the DR function for the CATE is linear (but not quadratic) in the debiasing terms, in which outcome regression residuals are reweighted by the inverse of propensity scores (also known as augmented residuals), the CLATE score involves a cross-product between the DR estimators of the reduced form and first-stage effects. Consequently, the CLATE moment condition contains both $\Theta_{X,z}$ and its square, reflecting the bilinear structure of the CLATE, which depends on the ratio of two conditional effects. This introduces second-order terms in the score but does not alter Neyman orthogonality, as the influence of the nuisance parameters still cancels out through the residual orthogonality conditions $E[Y - m_{w,X,z}\mid X,Z,W] = 0$ and $E[D - r_{w,X,z}\mid X,Z,W] = 0$. In this sense, the CLATE score extends the construction of scores based on squared differences in regression functions to a setting where both the numerator (reduced form) and denominator (first stage) of the parameter of interest must be debiased simultaneously.
As for Assumption (ref) in the CATE case, it is worth noting that Assumption (ref) may hold for two reasons: either CLATEs do not depend on unobservables, or the distribution of unobservables is stable across experiments. To illustrate, consider the outcome model in equation (ref) together with a threshold-crossing treatment model
where $I\{\ \cdot\}$ is the indicator function that is equal to one if its argument is satisfied and zero otherwise, $\lambda$ and $\eta$ are unknown functions, and $V$ are unobservables affecting the treatment.
We note that the threshold-crossing model for treatment assignment in equation (ref) both implies and is implied by treatment monotonicity, as shown in Vy02. Regarding effect heterogeneity, the unobservable $V$ may be arbitrarily associated with $U$ under our IV assumptions, so that heterogeneity of treatment effects in $U$ generally also induces heterogeneity with respect to $V$ and hence across compliance types, defined by whether $I\{\lambda(1,X)>\eta(V)\}=1$ and $I\{\lambda(0,X)>\eta(V)\}=0$. Therefore, if it can be assumed that unobservables differ across $Z$, then satisfaction of Assumption (ref) points to homogeneous effects. This, in turn, implies that treatment effects do not depend on compliance behavior - which is itself a function of unobservables - conditional on $X$, an assumption discussed in AnFe2010 and AronowCarnegie2013:
An important implication of this assumption is that it allows extrapolating the CLATE to the entire population, since under Assumption (ref) the CLATE coincides with the CATE. In other words, the identified effect is no longer local to compliers, but represents the average conditional effect for the full population.
Further, alternative identifying assumptions can be considered when panel data (or also repeated cross sections) are available, in which outcomes are observed both before and after the introduction of treatment. To this end, we introduce time index $t \in {0,1}$, where $t=0$ refers to the pre-treatment period and $t=1$ to the post-treatment period, to denote by $Y_t$ and $Y_t(d)$ the outcome and the potential outcome (given $D=d$) at time $t$, respectively. This setting permits effect identification based on the parallel trends assumption, which requires conditional independence in outcome trends rather than in outcome levels (as imposed by Assumption (ref)). A set of sufficient assumptions for identifying the conditional average treatment effect on the treated (CATET) in panel data based on the difference-in-differences (DiD) approach is the following, see, e.g., Abadie2005; Lechner2010:
The first condition in Assumption (ref) formalizes the conditional common trends assumption: given (presumably exogenous) covariates $X$ and experiment $Z$, no unobserved factors simultaneously affect both treatment assignment and the trend of mean potential outcomes under non-treatment. In DiD settings, it is worth noting that in the context of DiD, multiple experiments $Z$ may for instance correspond to multiple treated regions observed within the same dataset. The second condition rules out average anticipation effects among the treated, conditional on $X$. It requires that treatment status $D$ does not causally influence pretreatment outcomes in expectation of the treatment to come. The third line imposes a specific common support condition for identifying the CATET, requiring that for every covariate profile $X$ and experiment $Z$ observed among the treated, there also exist some untreated observations with the same $(X,Z)$.
When replacing $Y$ by the outcome difference $Y_1-Y_0$ in the definitions of the conditional mean outcomes $\mu_{D,X,Z}$ introduced in Section (ref), the null hypotheses (ref) and (ref), as well as the score function (ref), can be redefined to evaluate effect heterogeneity across CATETs in different experiments. A natural question is whether these effects can be extrapolated to the total population, i.e., whether the CATET coincides with the CATE. In general, this is not the case because the parallel trends condition in Assumption (ref) is only imposed for the untreated potential outcomes, such that identification is restricted to the treated group. In particular, treatment effects may differ with respect to time-invariant confounders that are allowed to differ between treated and untreated units. However, if experiments differ in such time-invariant confounders, and effect homogeneity across experiments is not rejected, this suggests that treatment effects do not depend on them. In this case, the CATET coincides with CATE, as expressed in the following assumption:
The following structural model illustrates the role of time-invariant unobservables. Let
where $\kappa_t(D,X)$ is an unknown, time-varying function of covariates $X$ and treatment $D$, $\eta(D,U)$ is a time-invariant function of unobservables $U$ that may interact with treatment, and $\varepsilon_t$ is an idiosyncratic, time-varying error. For the potential outcome under non-treatment,
it follows that differencing across time, $Y_1(0)-Y_0(0)$, eliminates $\eta(0,U)$ due to its additive separability. Hence, the parallel trends condition holds with respect to $Y_t(0)$ conditional on $X$, even if the distribution of $U$ differs between treatment groups. However, treatment effects may still vary across groups, since arbitrary interactions between $D$ and $U$ are allowed in $\eta(D,U)$. Now consider the alternative model
which rules out such interactions and implies additive separability of $U$ and $D$. In this case, treatment effects are homogeneous in $U$, as in classical linear panel regression models. Therefore, if $U$ plausibly varies across experiments $Z$ but CATETs are found to be constant across $Z$ (and thus across distributions of $U$), this provides evidence in favor of Assumption (ref), which justifies extrapolating effects identified for the treated (CATET) to the entire population (CATE).
This section describes a simulation study to investigate the finite sample behavior of our proposed test of homogeneity of CATEs across experimental and observational sites. We first consider comparisons across experiments and and base our simulations on the following data generating process (DGP):
Where the outcome $Y$ is a function of the treatment $D$, $X$ the covariates, $Z$ is an indicator of an experimental site and $U$ denotes the error term. In the experimental design, $D$ is randomly assigned based on a Bernoulli distribution with probability $q$ and Assumption (ref) (conditional independence) holds by construction. $X$ is a vector of covariates of dimension $p$, drawn from a multivariate normal distribution with zero mean and covariance matrix $\Sigma$. In this specification, $\Sigma$ equals the identity matrix, implying that all covariates are independent and have unit variance. $Z$ is an indicator of an experimental (or observational) site, generated independently from a Bernoulli distribution with probability $\pi$. The coefficients $\beta$ determine the impact of the covariates $X$ on $Y$. Finally, $U$ is a random and normally distributed error term.
In the observational design, the element which changes is the treatment assignment which is no longer randomized:
Thus, we consider the case where treatment $(D_{obs})$ depends on covariates $X$ and the error terms $U$ and $V$. Where $\rho$ determines the strength of confounding in the observational sites. At the same time, the parameter $\delta$, from the outcome equation, governs the degree of effect heterogeneity across experimental and/or observational sites. Consequently, when $\delta=0$, treatment effects are constant across sites, while $\delta \neq 0$ induces heterogeneity in CATEs across sites, indexed by $Z$. Likewise, when $\rho=0$, there is no confounding in the observational sites, while $\rho \neq 0$ introduces confounding.
We implement a cross-fitted, doubly robust (DR) score test based on double difference score in (ref), adapted from apfel2024learningcontrolvariablesinstruments to test whether CATEs are homogeneous across sites $Z$. For each $Z$, we estimate conditional means and joint propensities by a penalized lasso regression using five fold $k=5$ cross fitting with default parameters as in the glmnet package in R. We then build DR residuals and combine them into a double difference score across individual sites $z$. To ensure overlap, we trim the estimated conditional probabilities below $0.05$ and above $0.95$.
The overall test statistic is computed as the sample mean of the individual scores over the retained sample after trimming. In addition, the standard error is obtained from the score variance scaled by the sample size. Finally, a normal approximation is used to compute $p$-values. The simulation scenarios vary in several dimensions. We consider three sample sizes, $n = 500, 2000$ and $8000$. In our main specification, we set \( k = 5 \), \( l = 2 \), \( p = 100 \), use Lasso as the machine learner, and fix the trimming threshold at \( \varepsilon = 0.05 \). We perform $R=1000$ Monte Carlo replications. To assess size, we impose the null of cross-site homogeneity by setting $\delta=0$. To assess power, we introduce cross-site heterogeneity by setting $\delta=1$. In the mixed design, we further allow for unobserved confounding in observational sites by setting $\rho\in\{0,0.5\}$, so that we study the performance of the test under both unconfounded and confounded observational assignment.
On the one hand, when $\delta=0$ where we impose homogeneity of CATEs across sites, the rejection rate of the test should approach the nominal significance level (5%) as $N$ increases, reflecting correct size of the test. On the other hand, when $\delta=1$ where we impose heterogeneous CATEs across sites, the rejection rate should increase with $N$, demonstrating the correct power of the test. We assess the performance of the proposed test using several summary measures. Across $R=1000$ Monte Carlo replications, we report the average estimate ($\hat{\theta}$) based on (ref), its standard deviation (std), and the average estimated standard error (mean se). We report empirical rejection rates at the 5% level, interpreted as the size when $\delta=0$ and power when $\delta\neq 0$, as functions of $N$ and $\delta$. Finally, we report the effective sample size as a function of the trimming rate.
Table (ref) reports the simulation results based on our main specification when $\delta = 0$ across experimental sites only. That is, when the null hypothesis of homogeneous CATEs across experimental sites is true. The average estimate $\hat\theta$ of the test decreases toward zero as the sample size increases, consistent under the null. The standard deviation and the average standard error decrease by roughly half when the sample size $N$ quadruples, indicating that the estimator is root-$N$ consistent. The empirical rejection rate is slightly above the nominal 5%. However, it moves towards the correct levels as $N$ increases. Lastly, the effective sample size is close to the nominal sample size $N$ in all specifications, suggesting that trimming is limited and weights are stable. Overall, the results indicate that our test behaves correctly under homogeneous treatment effects across experimental sites and is well calibrated under the null.
Table (ref) reports the simulation results based on our main specification when $\delta = 1$, where CATEs vary across experimental sites and the null hypothesis of homogeneous CATEs is false. In this case, the test should reject with high probability. The average estimate $\hat\theta$ becomes increasingly negative as the sample size increases. Both the standard deviation and the mean estimated standard error of the estimator decrease roughly by half when the sample size quadruples, again consistent with root-$N$ convergence. The empirical rejection rate rises sharply with sample size increasing from 79% to 100% for larger samples. Finally, the effective sample size remains close to the nominal sample size for all $N$ in all specifications. The results show that our test has strong power to detect violations of effect homogeneity across sites.
We now consider the setting which combines experimental and observational sites in the absence of confounding where $\rho = 0$. This mixed design mirrors many empirical applications in which treatment is randomized in some sites but not in others, with the latter relying on observational variation. This allows us to examine whether our method can distinguish violations of homogeneity from violations of unconfoundedness. When experimental sites indicate homogeneous CATEs, systematic differences between experimental and observational CATEs provide evidence against the validity of the observational identification strategy.
Table (ref) reports simulation results for the mixed experimental–observational design under the null hypothesis of homogeneous treatment effects across sites, $\delta = 0$, and no unobserved confounding in the observational sites, $\rho = 0$. Consequently, the identifying assumptions hold, and the test should reject at the nominal significance level. The mean estimate $\hat{\theta}$ moves toward zero as $N$ increases, consistent with the null. The standard deviation and the average estimated standard error decline at the expected rate. The empirical rejection rate is above the nominal 5% level in smaller samples but it declines with sample size. Finally, the effective sample size remains close to the nominal sample size in all designs. Overall, the results indicate that our test behaves as expected when experimental and observational sites are combined and the observational identifying assumptions hold, although it exhibits some over-rejection in smaller samples.
We next consider a mixed design which combines experimental and observational sites, but now we allow for unobserved confounding in the observational sites captured by $\rho = 0.5$. This value introduces strong confounding in the observational sites. In this setting, the observational identifying assumptions fail, so discrepancies between experimental and observational CATEs are driven by confounding rather than by true effect heterogeneity. As a result, the test may experience distortions in its size, reflecting sensitivity of the test to violations of the identifying assumptions in the observational sites.
Table (ref) reports the results for the mixed experimental-observational design under homogeneous treatment effects across sites where $\delta = 0$ and strong unobserved confounding in the observational sites where $\rho = 0.5$. The mean estimate $\hat{\theta}$ decreases at a slower rate than under the scenario of no confounding as the sample size increases. However, both the Monte Carlo standard deviation and the mean estimated standard error shrinks at the expected root-$N$ rate. The effective sample size also remains close to the nominal sample size. Moreover, the rejection rates increase substantially even for moderate sample sizes. The results suggest that, as $N$ grows, our test increasingly detects differences in CATEs between experimental and observational sites which are driven by unmeasured confounding rather than by true treatment effect heterogeneity across sites.
Table (ref) reports the results for the mixed experimental-observational design under heterogeneous treatment effects across sites where $\delta = 1$ and there is no unobserved confounding where $\rho = 0$ in the observational sites. Across sample sizes, the mean test statistic $\hat{\theta}$ is negative and becomes slightly more negative as $N$ increases. The standard deviation and the estimated mean standard error decrease approximately at the root rate $N$. Consistent with this, the rejection rate rises as the sample size increases and reaches essentially one for $N \ge 2000$. The effective sample size remains close to the nominal $N$ in all cases. Overall, the simulation results indicate that our test has high power to detect cross-site heterogeneity when identification is valid in both experimental and observational sites and its power increases rapidly with sample size.
Finally, Table (ref) reports the results for the mixed experimental-observational design under heterogeneous treatment effects across sites where $\delta = 1$ and unobserved confounding in the observational sites where $\rho = 0.5$. The results show that the test statistic becomes increasingly negative as the sample size increases while the standard deviation and average standard error decrease at root $N$. Regarding power, the results show that the test has lower power in small samples, while the rejection rates increase sharply as the sample size grows. This indicates that strong confounding can weaken the detection of effect heterogeneity in finite samples, but the test regains power for larger sample sizes. Finally, the retained sample size is close to the original sample size. Importantly, because effect heterogeneity and confounding are present simultaneously in this scenario, the test detects differences in treatment effects between experimental and observational sites, but it cannot attribute that difference uniquely to true treatment effect heterogeneity versus violations of the identifying assumptions in the observational sites. Overall, the simulations show that our proposed test is well behaved when its identifying assumptions hold and becomes increasingly informative as the sample size grows.
We illustrate our method using data from the International Stroke Trial (IST), a large multi-centre randomized controlled trial in acute ischaemic stroke conducted by the IST Collaborative Group Sandercock2011. The IST investigated whether early administration of aspirin, heparin, both, or neither affects clinical outcomes after stroke. Patients were eligible if they had a clinical diagnosis of acute ischaemic stroke within 48 hours of symptom onset and had no clear indication for, or contraindication to, either treatment. After a CT scan to support the diagnosis, clinicians contacted a central randomization service that recorded baseline characteristics and returned the assigned treatment.
The dataset contains anonymized individual-level information on 19{,}435 patients treated in 467 hospitals across 36 countries. It includes baseline characteristics, clinical status at randomization, short-run outcomes measured at 14 days, and follow-up outcomes at six months. The primary outcome of interest of the trial was death or dependency in daily living six months after randomization. Our empirical analysis focuses on the randomized assignment to aspirin. We define the treatment indicator $D$ as assignment to aspirin and the outcome $Y$ as an indicator equal to one if the patient is dead or dependent at six months, and zero otherwise. We restrict the sample to patients with non-missing information on treatment assignment and the six-month outcome. Additionally, for comparability, we focus on observations from the main trial and discard observations from the pilot trial.
Table (ref) summarizes the sample construction. Starting from the full IST dataset, we first observations from the pilot phase to focus on the main trial for comparability reasons. We then restrict the sample to patients with non-missing treatment assignment and six-month outcome. Finally, we impose a minimum sample size requirement at the site level and retain only countries with at least 50 observation. The final analysis sample contains $N=18{,}189$ patients from 31 countries listed in Table (ref). As baseline covariates $X$, we use pre-treatment patient characteristics measured at randomization: age, sex, systolic blood pressure, indicators for baseline level of consciousness (fully alert, drowsy, unconscious), and whether a CT scan was performed before randomization. These covariates capture clinically relevant differences at baseline health between patients.
Table (ref) summarizes baseline covariate balance between treated and control patients using standardized difference in means (SMDs). The treated and control groups are very similar across all characteristics as all SMDs are below conventional thresholds for meaningful imbalance. In particular, age, systolic blood pressure, sex, pre-randomization CT use, and baseline consciousness status are nearly identical across treatment arms. These patterns support a successful treatment randomization in the IST study. In addition, treatment assignment is also balanced within each country, although sample sizes within countries vary substantially (see Figure (ref) and Table (ref)). In this context, we take countries as experimental sites, indexed by $Z$, and apply our test of effect homogeneity between these countries. This setting is well suited to our approach because a common randomized protocol was implemented across all participating countries. However, clinical environments and baseline risk may differ between sites, as can be seen in Figure (ref).
Figure (ref) displays outcome rates by country. The outcome rates vary markedly across sites, ranging from 0.324 to 0.802, which likely reflects differences in baseline risk, case mix, and clinical practice across countries. This heterogeneity in outcome levels motivates testing whether the treatment effects are homogeneous across countries, specifically if the conditional effect of aspirin on death or dependency is stable between countries. We therefore apply our proposed test of CATE homogeneity across multiple experimental sites to the IST dataset. The estimand is the average treatment effect of randomized aspirin assignment on six-month death or dependency, adjusting for baseline covariates. In the spirit of our framework, failing to reject homogeneity indicates no systematic interactions between treatment and unobserved determinants of outcomes in the mean effect and supports generalizing the estimated effect across countries. In contrast, rejecting homogeneity suggests that the effect varies across sites in ways not captured by observed covariates, consistent with treatment–unobservable interactions and/or other forms of site-level heterogeneity.
Table (ref) reports the results of our test of CATE homogeneity across countries, implemented with two trimming thresholds, $\varepsilon\in{0.05,0.10}$. For the baseline threshold $\varepsilon=0.05$, the test rejects homogeneity at the 5% level ($p=0.046$), providing marginal evidence that the conditional effect of aspirin may vary across countries. Increasing trimming to $\varepsilon=0.10$ yields a much smaller test statistic and a large $p$-value ($p=0.706$), so we no longer reject homogeneity. The shift in inference is accompanied by a sharp decline in the effective sample size.
The results show the impact that trimming and therefore, overlap can have on the conclusions. Importantly, in this context, the sensitivity of the results to trimming is not due to the limited overlap in treatment assignment, as aspirin is randomized within each country and the estimated propensity scores are tightly concentrated around $0.5$. Rather, it reflects limited support for certain country covariate combinations. In some countries, specific covariate profiles are rare and/or the country sample size is small, resulting in estimated site-specific propensity scores that can be very low. This poses a problem because the estimation weights are based on the inverse of the propensity scores, so small scores translate into very large weights. As a result, a small number of observations in sparsely supported regions of the country-specific covariate distribution can receive extreme weights and disproportionately influence the test statistic. Trimming prevents this by excluding such cases with extreme propensity scores. With stricter trimming that focuses on regions with improved common support, the test does not reject the null hypothesis of homogeneous treatment effects across countries.
In this work, we introduced a framework for testing the homogeneity of conditional average treatment effects (CATEs) across multiple experimental and observational sites. The proposed test is built on a Neyman orthogonal score that extends apfel2024learningcontrolvariablesinstruments to a double difference setting. Under specific regularity conditions (in particular, $o(n^{-1/4})$ convergence rates for the nuisance estimators) and with cross-fitting, the resulting estimator is $\sqrt{n}$-consistent and asymptotically normal. We also showed how the same logic carried over to settings with alternative identification strategies, such as instrumental variables and panel designs with parallel trends.
The simulation study indicated that the test is well behaved when the identifying assumptions hold and becomes increasingly informative as sample size grows. In randomized multi-site designs, the test has the correct size under homogeneity and has high power against heterogeneity. In mixed designs that combine experimental and observational sites, the test rejects systematically when unobserved confounding is present in the observational sites, even when treatment effects are homogeneous, highlighting the usefulness of the test as a diagnostic for flagging potential confounding.
We then illustrated the approach using data from the International Stroke Trial, treating countries as sites and testing whether the conditional effect of randomized aspirin assignment on six month death or dependency is homogeneous across countries. With more permissive trimming, we reject homogeneity, while with stricter trimming we do not reject. This sensitivity likely arises because some countries have very few observations for certain patient profiles. With a more permissive trimming rule, these rare profiles can receive very large weights and therefore have a disproportionate impact on the results. However, with stricter trimming, we exclude those poorly represented cases, and the analysis relies on patient profiles that are more common within each country.
Overall, the proposed framework provides a practical and flexible tool for assessing homogeneity of CATEs across experimental and observational data. The test helps researchers to diagnose confounding as well as to evaluate the internal and external validity of their estimates. This is increasingly valuable as researchers now often have access to both randomized trials and rich observational datasets, but lack principled ways to determine when estimates from these sources can be compared, combined, or extrapolated across settings.
\singlespacing \setlength\bibsep{0pt}
\FloatBarrier
\addcontentsline{toc}{section}{References}