Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
91,261 characters · 10 sections · 90 citation commands
Decomposition and Interpretation of Treatment Effects in Settings with Delayed Outcomes
KEYWORDS: delayed outcomes, treatment effects, regression, total effects, direct effects, downstream engagement, sign preservation.
JEL classification codes: C12, C14.
\thispagestyle{empty} \setcounter{page}{1}
We study settings where the analyst is interested in identifying and estimating an average direct causal effect of a binary treatment on an outcome, and the treatment status is determined in the context of a randomized controlled experiment or an observational study under conditional independence assumptions. We focus on settings where the outcome of interest does not get immediately realized after treatment assignment, a feature that is ubiquitous in empirical settings, including the study of long-run effects in economic history voigtlander/voth:12,angelucci/etal:22, the analysis of long-term valuation in industry settings jain2002customer,chen:2021, and randomized experiments in economics and other social sciences Beaman/etal:2013,Moderna. The delay in the realization of the outcomes creates a time window between the treatment assignment and the realization of the outcome that, in turn, opens up the possibility for other observed ex-post actions to take place before the outcome is finally realized; see Figure (ref) for a graphical representation. In this context, we study the interpretation of several popular estimands that arise from running regressions of the outcome on the treatment and different ways of “controlling” for the other actions. We emphasize that our use of the term “regression” refers to a linear projection that is free of any modeling assumptions on potential outcomes or conditional means. Some of these estimands are not only popular in the economics literature, see, e.g., fagereng2021wealthy,heckman2013understanding,chernozhukov/etal:21joe, but are also widely used across other sciences, like psychology, epidemiology, biostatistics, and political science, as shown by the large number of citations associated with the regression approach popularized by baron1986moderator. For each of these estimands, our results present a decomposition that facilitates their interpretation in terms of ceteris paribus effects of the treatment on the outcomes, indirect effects caused by the other actions, and selection terms; and provide a framework that allows us to clarify under what type of conditions the practice of “controlling” for the presence of other actions leads to estimands that admit the desired interpretation.
The main findings of this paper can be grouped into three sets of results. First, the standard practice of studying estimands that arise from a regression of an outcome on the treatment, with or without “controlling” for the other actions in such regressions, does not generally satisfy what we call strong sign preservation. Strong sign preservation, formally characterized in Definition (ref), is satisfied when an estimand that intends to measure a ceteris paribus (or direct) causal effect of a treatment on an outcome is positive when the effect of the treatment on the outcome is positive conditional on all possible values of the other actions. Failure to satisfy strong sign preservation introduces a Simpson’s paradox-like sign reversal where the estimands may be negative even when the treatment positively affects the outcome for any possible combination of other actions. Second, the most popular estimand that linearly controls for the other actions in the regression, and that we label the long regression, does not generally provide benefits relative to the short regression that includes no controls whatsoever. More concretely, while neither the short nor the long regression satisfies strong sign preservation, the estimand associated with the long regression admits a decomposition in terms of weighted averages of well-defined causal effects but where the weights could potentially be negative. This feature introduces yet another source that may separate the sign of the estimand from the sign of ceteris paribus causal effects. Notably, this feature also occurs when the regression includes interaction terms between the treatment and the other actions. Perhaps our most salient result is the one in Theorem (ref), which shows that the long regression delivers easy-to-interpret results if and only if the other actions are all binary and mutually exclusive random variables. Finally, while non-parametric identification of the effects is straightforward under the stronger form of our assumptions and follows directly from a saturated regression, we also show that a linear regression that properly controls for other actions through complete stratification produces estimands that satisfy strong sign preservation. We term this the “strata fixed effects regression” due to its link to the practice of including strata fixed effects in randomized controlled trials with covariate adaptive randomization (see bugni/canay/shaikh:2018,bugni/canay/shaikh:2019).
The decompositions we derive for each of the five regression-based estimands can be interpreted as a partition into “direct” and “indirect” effects (and possibly “selection” effects depending on the assumptions), and so our results are linked to the vast literature on mediation analysis, see, e.g., baron1986moderator, pearl:01, robins:03, kaufman2004critique, kaufman2009gilding, imai2010, vanderweele/etal:14, and Remark (ref) for a discussion. See vanderweele-book:15 for a book-level treatment. However, as opposed to the literature on mediation that studies the type of assumptions that would identify the causal effects of the so-called mediators, which in our context would simply be the other actions taken before the outcome is realized, here our goal is not to identify these indirect effects but rather to gain a better understanding of how to properly interpret certain popular estimands of the effect of the treatment on the outcome without the aid of functional form assumptions.
Beyond the literature on mediation analysis, our paper also connects to several strands of research in econometrics and biostatistics. First, we are not the first to highlight the importance of distinguishing between “direct” (or “partial”) and “total” causal effects in econometrics. Early discussions include those in Manski:97 and heckman:00; see also Remark (ref). Second, our results relate to a large body of research that interprets regression-based estimands in various settings, often characterizing them as weighted averages of causal effects of interest—sometimes involving negative weights. Examples include the literature on two-way fixed effects in difference-in-differences settings DeChaisemartin/xavier:20,goodman-bacon:21,callaway/santanna:21,sun/abraham:21,borusyak/jaravel:22; on local instrumental variables and marginal treatment effects canay/etal:2023; and on contamination bias when conditioning on covariates angrist:98,goldsmithpinkham/etal:22. Our paper is also connected to the literature on design-based analyses of regression estimands in factorial experiments zhao/ding:22. These connections are primarily algebraic—stemming from shared properties of least squares—though the questions we address and the main takeaways of our analysis are distinct. Third, our results on the failure of strong sign preservation for popular estimands resonate with the “surrogate paradox” in the surrogacy literature, where a treatment may have a positive effect on a surrogate, and the surrogate may be positively correlated with the outcome, yet the treatment effect on the outcome may be negative; see chen/etal:07 and vanderweele:13. Finally, the decomposition we derive for the short regression mirrors those developed in the literature on causal interactive effects—where units may be subject to multiple types of exposures (e.g., genetic and environmental)—as in robins1992identifiability and vanderweele:14. Our focus on a broader range of regression estimands, however, sets our work apart.
The remainder of the paper is organized as follows. Section (ref) introduces the basic notation. Section (ref) defines the main concepts we use throughout the paper, including partial causal effects, direct causal effects, and strong sign preservation. Section (ref) introduces the five estimands we study and then presents the main results on how each of these estimands admits different decompositions into direct, indirect, and selection effects. Finally, Section (ref) concludes.
Consider a setting where $Y$ denotes the observed outcome of interest, and the actions taken by individuals or units under study are divided into a “main” action of interest, denoted by $D$, and “other” actions, denoted by $A$:
Let $X$ represent other observed covariates, which include features beyond actions.
All actions are assumed to be discrete. The main action, $D$, is further assumed to be binary, i.e., $\mathcal D \equiv \{0,1\}$. The other actions, $A$, form a $K$-dimensional vector taking values in \( \mathcal A \equiv \{a=(a_1,\dots,a_{K}): a_j\in \mathcal A_j \text{ for } j=1,\dots,K \} \), where each $ \mathcal A_j$ is a finite set. For notational convenience, we assume $ \mathcal A_j \subseteq \mathbb N$, $0 \in \mathcal A_j$, and that when $|\mathcal A_j| = 2$, we have $ \mathcal A_j = \{0,1\}$. These additional restrictions simplify the discussion and expressions but are not required for our results.
The setting we study in this paper is one with the following characteristics. First, the analyst controls the action of interest $D$ via either a randomized controlled experiment or, alternatively, by an exogeneity assumption like selection on observables. We therefore alternatively call this action the “treatment”. Second, the outcome $Y$ is not instantaneous and takes some time to be realized within the timeline of the experiment. In the period between treatment assignment and outcome realization, the other actions in $A$ are chosen by the units—these may be modeled as conditionally exogenous in some applications, or left unmodeled altogether. Figure (ref) illustrates the setting. Below we describe some empirical applications in economics, social sciences, and industry that naturally fit into this setting.
The first class of applications that fit our framework is the literature that studies long-run outcomes in economics. There, interest typically lies in a treatment that happened several years in the past (oftentimes hundreds of years ago) on some outcome of interest in more recent times. For example, voigtlander/voth:12 study the effect of the existence of Black Death pogroms in 1349 ($D$) on the level of anti-Semitism in Nazi Germany in 1920s ($Y$) at the city level. Other actions that are included in the analysis include city population in the 1920s ($A$), among other variables that are determined closer to the realization of the outcomes. Other examples include nunn:08 and angelucci/etal:22, among many others. In these applications, it is common to argue that the treatment is exogenous invoking selection on observables assumptions or, in some cases, relying on instrumental variables approaches. Our results are limited to settings exploiting conditional exogeneity.
The second class of applications arises in the analysis of long-term valuation in marketing and industry. In these settings, companies such as Airbnb, Uber, Google, and Microsoft operate platforms where users engage in a wide range of actions—such as making a booking, writing a review, subscribing to a service, or watching a video—and are interested in estimating the long-run value that a specific product or action generates for the firm. For instance, chen:2021 discusses how Airbnb measures the long-term value of user actions by tracking the revenue generated over a 365-day window. The example focuses on the effect of a guest making a booking ($D$) on long-term revenue ($Y$), while other intermediate actions—such as cancellations or writing reviews—are captured in the vector $A$. Although randomized experiments are often feasible for some actions, others cannot easily be randomized due to ethical, legal, or user experience concerns. In such cases, it is common to rely on conditional exogeneity (i.e., selection on observables) and focus on one action of interest at a time. Importantly, in these applications, the goal is often to identify a direct causal effect of the focal action ($D$) on the outcome ($Y$), holding other actions ($A$) fixed. This approach helps avoid the problem known as double counting—that is, attributing the same impact multiple times through correlated or causally related behaviors. For this reason, isolating the direct effect of $D$ is not merely of academic interest, but a practical necessity in how firms estimate the marginal contribution of individual features or interventions. Related examples and discussions can be found in Lewis/etal:11 and xu/etal:2015, who discuss how correlated consumer behaviors—termed “activity bias”—can lead to substantial overestimation of advertising effectiveness.
The third class of applications includes clinical trials and randomized experiments with a follow-up period between the treatment assignment and the realization of the outcome of interest. There, researchers typically fully control the assignment of the main treatment of interest but are unable to restrict behavior in the follow-up period. For example, the clinical trial run by Moderna to study the efficacy of the Moderna COVID-19 vaccine against SARS-CoV-2 infections. Participants in the study were randomized to Immediate Vaccination Group 1 (receiving the Moderna COVID-19 Vaccine on Day 1 and Day 29) or Standard of Care Group 2, with vaccination given at months 4 and 5. During the months following vaccinations, participants received visits that checked for infections and could include blood collection, nasal swabs, SARS-CoV-2 screening, COVID-19 symptom checks, and questionnaires. In this example, $D$ would be an indicator of whether the participant received a vaccine, $Y$ would be an indicator of whether the participant got infected within the 4 months of the study, and $A$ would include other actions taken by the participants that could affect infection rates, like whether the participants wear masks in public, whether the participants avoid large gatherings, etc.\footnote{In this paper we abstract away from spillover effects between individuals, which could be relevant in the context of this and other examples we describe.} Another example is Beaman/etal:2013, who conducted a randomized field experiment that provides free fertilizer to female rice farmers in Mali to measure how they choose to use the fertilizer and the overall impact on profitability. The authors distributed the fertilizer in May 2020 and conducted two follow-up surveys, one in August 2020 and one in December 2020, right after the harvest. In this example, $D$ would be an indicator of whether the farmer received free fertilizer, $Y$ would be a measure of output like crop yield or just profits, and $A$ would include all relevant complementary agricultural inputs, such as labor, herbicides, and water usage. Due to its simplicity, we use this farming example to illustrate the concepts we define in the next section.
We denote potential outcomes by $Y(d,a)$ and their expectation by
We also introduce the concept of a pooled potential outcome to isolate the counterfactual outcome associated with the main action of interest (the treatment),
where $A(d)$ denotes potential outcomes for the actions $A$ as a function of the treatment $d$. Finally, the observed outcome $Y$ is related to potential outcomes by the relationship
In the next section, we define the type of causal effects we are interested in this paper, as well as the assumptions that we invoke to interpret the decompositions we derive for a variety of estimands.
We start by discussing the type of counterfactual treatment effects that could interest the researcher in the canonical setting where $D$ is binary. Viewing $Y(d,a)$ as a function of two types of actions immediately suggests that there could be total effects, direct effects, and indirect effects, all of which may or may not be of interest in the context of a concrete application. Understanding the variety of causal effects that one could describe, in turn, will help us provide representations and interpretations of commonly used target parameters, like the average treatment effect (ATE), in terms of these types of causal effects. We start with what is perhaps one of the most natural types of ceteris paribus effects in Definition (ref).
Definition (ref) defines an average partial causal effect of the main action as a mean comparison that keeps the value of the other actions unchanged in both states of the comparison (ceteris paribus). In the farming application of Beaman/etal:2013, it would capture the average causal effect of using fertilizer on the crop yield, while keeping other inputs, like labor, herbicides, and water usage, constant in the counterfactual comparison.
The ceteris paribus effect in Definition (ref) could also be defined conditional on certain events or subpopulations. To account for this, we also consider the concept in Definition (ref) conditional on some set $\Omega$, i.e., $E[Y(1,a)-Y(0,a) \mid \Omega ]$, where $\Omega$ is a function of $(D,A,X)$. For example, $\Omega = I\{D=1\}$ would lead to an average partial causal effect on the treated and $\Omega = I\{X=x\}$ would lead to an average partial causal effect for units with covariates $x$.
The definition of a partial causal effect for the main action delivers a potentially different causal effect for each possible value of the other actions or, alternatively, provides a collection of partial causal effects indexed by $a\in\mathcal A$. While the goal could just be to identify such a collection of effects, in many settings it may be natural to aggregate this collection of partial effects in a way that summarizes the effect of the main action on the outcome of interest. The following definition defines a direct causal effect as any weighted average of partial causal effects.
In the context of our farming example with $A$ only capturing low and high water usage for simplicity, the parameter (ref) combines the average causal effect of using fertilizer on the crop yield for units with high water usage, say $A=1$, and units with low water usage, say $A=0$. Average direct causal effects could also be defined conditional on a set $\Omega$. The definition does not determine how the groups are weighted, but it requires that no group is assigned a negative weight. In this sense, any average direct causal effect satisfies strong sign preservation, as defined below.
In our farming example with $A$ only capturing low and high water usage, strong sign preservation of a parameter $\Delta$ implies that whenever fertilizers improve the expected crop yield both for units with high water usage and units with low water usage, $\Delta$ should be positive as well. As the name suggests, strong sign preservation does not allow for the possibility of what it is typically referred to as sign reversal, understood as a situation where $\Delta<0$ when $\mu(1,a)-\mu(0,a)>0$ for all $a\in \mathcal A$.
A premise of this paper is that researchers employ regression-based estimands to obtain an average direct causal effect, as defined in Definition (ref). That is, the primary interest is not in identifying indirect effects—capturing the impact of $D$ on $Y$ through the actions $A$—nor in identifying the total effect, which allows the actions $A$ to vary across treatment groups. While such alternative effects may be of interest in certain applications, our aim is to clarify when the estimands described in the next section can be interpreted in line with Definition (ref).
The decompositions we present in this paper are purely algebraic and, in their most general form, do not rely on any identification assumptions. However, to interpret the terms in these decompositions as causal effects—or to simplify their expressions—it is useful to introduce assumptions that both facilitate interpretation and connect our framework to assumptions commonly used by practitioners and related work in the literature.
We begin with a baseline assumption that allows for the identification of a total effect of $D$ on $Y$, i.e., $E[Y(1)-Y(0)]$. This assumption is standard in the literature on treatment effects in settings that do not explicitly account for the presence of the other actions $A$.
Assumption (ref) can be justified by the design of a randomized controlled experiment Beaman/etal:2013 or by relying on a sufficiently rich set of covariates—as is often the case in industry applications—to make the conditional independence assumption credible chen:2021. Three points are worth emphasizing. First, as previously noted, this assumption is not required to derive our decompositions, but rather to aid in their causal interpretation. Second, as noted by a referee, one could replace Assumption (ref) with $D \perp (Y(d,a), A(d)) \mid X$ for all $(d,a) \in \mathcal{D} \times \mathcal{A}$. Although this yields equivalent interpretations, we favor Assumption (ref) as it more closely aligns with applications where other actions are not explicitly modeled. Third, while Assumption (ref) suffices to identify the average treatment effect of $D$ on $Y$, it does not generally support a causal interpretation of estimands intended to capture ceteris paribus or “direct” effects of $D$ when $A$ is also present. For that purpose, we consider an alternative assumption that grants conditional independence of $Y(d,a)$ with respect to both $D$ and $A$ given $X$.
Assumption (ref) reframes the problem as one involving multiple (conditionally exogenous) treatments. Three comments are worth pointing out. First, this assumption is strong in the sense that it enables nonparametric identification of the full mean response function $\mu(d,a)$, as discussed in Section (ref). In fact, it is equivalent to Assumptions (i) and (ii) in vanderweele/etal:14, which are sufficient for identifying controlled direct effects. Second, as noted by a referee, one could instead impose the stronger condition $(D, A(1), A(0)) \perp Y(d,a) \mid X$ for all $(d,a)$, which leads to the same interpretations. We favor Assumption (ref) as it aligns better with our focus on controlled effects. Third, while it may hold in settings with a rich set of covariates, it may be questionable when only $D$ is experimentally assigned and $A$ reflects behavioral responses to $D$.
To illustrate, in the agricultural production example, if fertilizer ($D$) is randomly assigned but farmers adjust water usage ($A$) in response to unobserved shocks that also affect yields—such as weather or soil quality—then Assumption (ref) would likely fail. By contrast, in many industry applications, it is common to condition on high-dimensional covariates and assume that all actions, including $A$, are conditionally exogenous. In these settings, $D$ is often designated as the action of interest not because it is fundamentally different from the other actions, but because of how it enters a business decision—for example, it may correspond to the action assigned to a particular team or product line. As such, the full vector $(D,A)$ may be plausibly treated as jointly exogenous given a rich enough set of observed variables.
Importantly, while Assumption (ref) permits identification of \( \mu(d,a) \), we show in later sections that this is still not sufficient for interpreting common regression-based estimands as average direct causal effects of \( D \) on \( Y \), as defined in Definition (ref). This underscores the fragility of such interpretations even under relatively strong identification assumptions.
In this section, we analyze five natural and highly popular estimands intended to capture treatment effects of $D$ on $Y$. For each of these estimands, we derive a decomposition in terms of parameters that can be labeled according to Definitions (ref) and (ref) and discuss under what assumptions they can be interpreted as intended. To keep our exposition as simple as possible and be able to zoom in on the type of concerns we intend to highlight, in what follows we abstract away from issues related to improper control of the covariates $X$. In other words, we ignore the role of the covariates in the type of regressions we consider. This could be interpreted as a situation where the covariates are discrete, and the regressions are viewed as within-cell regressions with cells given by $X=x$, or more generally, where the covariates have been already accounted for by other means, like clustering or via a partially linear model, among many possibilities. In particular, we do not consider the possibility that the analyst improperly controls for these covariates by simply including a linear term in $X$ in the regressions, as this would create additional problems to those discussed here; see goldsmithpinkham/etal:22 for a detailed treatment on the consequences of not properly controlling for confounders. We also re-iterate that our use of the term “regression” refers to a linear projection that is free of any modeling assumptions on potential outcomes or conditional means.
The first such estimand is the usual difference in means, which we write here as the slope coefficient $\Delta_{\rm short}$ in a regression (projection) of $Y$ on $D$ and a constant term,
where $E[UD]=0$ by properties of projections and $E[U|D]=0$ follows from $D$ being binary. We call this the short regression.
The second estimand is the slope coefficient $D$ in a linear regression of $Y$ on $D$, a constant term, and the $K$ actions $A_1,\dots,A_K$,
where $E[VD]=E[VA_j]=0$ by properties of projections. We call this the long regression.
The third estimand is the slope coefficient $D$ in a linear regression of $Y$ on $D$, a constant term, the $K$ actions $A_{1},\dots ,A_{K}$, and their interactions with $D$,
where $E[eD]=E[eA_j]=E[eA_jD]=0$ by properties of projections. We call this the long regression with interactions. Note that this is not a fully saturated regression in general, since the random variables $A_j$ are allowed to take arbitrary values in $\mathbb N \cup \{0\}$.
The fourth estimand is the slope coefficient $D$ in a regression of $Y$ on $D$ and a set of indicator functions for all the values that $A$ takes,
where $E[\nu D]=E[ \nu I\{A=a\}]=0$ by properties of projections. Note that this is a regression of $Y$ on $D$ with “strata fixed effects”, where the event $\{A=a\}$ defines a stratum for each value of $a$. As a result, we call this the strata fixed effect (SFE) regression.
The last set of estimands are the slope coefficients $\Delta_{\rm sat}(a)$, for $a\in\mathcal A$, in a saturated regression of $Y$ on a set of indicator functions for all the values that $A$ takes and their interactions with $D$,
where $E[\epsilon D I\{A=a\}]=0$ by properties of projections and $E[\epsilon|D,I\{A=a\}]=0$ follows from $D$ and $I\{A=a\}$ being binary for all $a\in\mathcal A$. We call this the saturated (SAT) regression.
The short regression is algebraically very simple, so we build up toward the main result introducing the main concepts and notation along the way. The other regressions, on the contrary, have more opaque derivations, and so in those cases, we first present the formal results and then discuss their interpretation.
The slope coefficient $\Delta_{\rm short}$ in (ref) equals $\Delta_{\rm short} = E[Y|D=1]- E[Y|D=0]$ by elementary arguments. If we define
and note that
we can decompose $\Delta_{\rm short}$ into the following three terms,
where
The three terms in the above decomposition for $\Delta_{\rm short}$ have a clear interpretation and show that there are two channels of endogeneity introduced by the fact that the other actions, $A$, take place in-between the treatment assignment and the realization of the outcome of interest. The term in (ref), $\Delta_{\rm dce}^{\rm s}$, captures an average direct causal effect on the treated, as in Definition (ref). Note that this term conditions on $\Omega = I\{D=1,A=a\}$ and so it is a conditional effect like those previously defined. The term in (ref), $\Delta_{\rm ind}^{\rm s}$, admits a clean interpretation for each value $a\in \mathcal A$ under additional assumptions we introduce below. Without additional assumptions, this term is a type of “indirect” effect that contains the product of the difference in conditional probabilities, $\pi_{1}(a)-\pi_{0}(a)$, and a term that confounds the average partial causal effect of $A$ moving from $0$ to $a$ on $Y$, with selection that arises from the distinct conditioning sets $\{D=0,A=a\}$ and $\{D=0,A=0\}$. Finally, the term in (ref), $ \Delta_{\rm sel}^{\rm s}$, is a selection term that captures the fact that $Y(0,a)$ may not be independent of $(D,A)$.
Three points are worth highlighting. First, the decomposition above does not rely on either Assumption (ref) or Assumption (ref). While Assumption (ref) (ignoring the $X$) ensures that
where \( Y(d) \) denotes the pooled potential outcomes defined in (ref), it is not sufficient to interpret \( \Delta_{\rm short} \) as an average direct causal effect or as a parameter satisfying strong sign preservation. In particular, the two endogeneity channels—\( \Delta_{\rm ind}^{\rm s} \) and \( \Delta_{\rm sel}^{\rm s} \)—in the decomposition of \( \Delta_{\rm short} \) may be positive or negative and, crucially, may cause \( \Delta_{\rm short} \) to have the opposite sign of \( \Delta_{\rm dce}^{\rm s} \). Second, these two endogeneity channels are conceptually distinct. The components in \( \Delta_{\rm ind}^{\rm s} \) reflect indirect pathways and are generally difficult to eliminate, while the selection term \( \Delta_{\rm sel}^{\rm s} \) becomes zero under Assumption (ref). Finally, under a given set of assumptions, the decomposition we derive for \( \Delta_{\rm short} \)—as well as for the other estimands analyzed in subsequent sections—is unique up to the choice of conditioning set (in our case, \( \Omega = \{D=d, A=a\} \)) and the normalization (here, the indirect effect is defined relative to \( \mu(0,0) \)). Once Assumption (ref) is imposed, the conditioning event becomes irrelevant, and the only remaining degree of freedom lies in the choice of normalization for the indirect effect.
Under Assumption (ref) the three terms entering the decomposition for $\Delta_{\rm short}$ simplify in the following way: $\Delta_{\rm sel}^{\rm s} = 0$ and
That is, the selection term $\Delta_{\rm sel}^{\rm s}$ is no longer present, and the direct effect $\Delta_{\rm dce}^{\rm s}$ and indirect effect $\Delta_{\rm ind}^{\rm s}$ are now a function of the unconditional expectations $\mu(d,a)$. Importantly, the term $\Delta_{\rm ind}^{\rm s}$ is still part of the decomposition since Assumption (ref) does not restrict how $A$ may affect outcomes, so that $\mu(0,a)-\mu(0,0)\ne 0$, nor does it affect how the main action may affect the other ones, so that $\pi_{1}(a)-\pi_{0}(a)\ne 0$. Aside from removing the term capturing selection bias, Assumption (ref) also delivers a clean interpretation of the indirect effects captured by $ \Delta_{\rm ind}^{\rm s}$. Each summand in $ \Delta_{\rm ind}^{\rm s}$ contains the average partial causal effect of $A$ moving from $0$ to $a$ on $Y$, $\mu(0,a)-\mu(0,0)$, multiplied by the difference $\pi_{1}(a)-\pi_{0}(a)$, which admits a causal interpretation of an average direct causal effect of $D$ on $A$ under the additional assumption $A(d)\perp D$.
We can interpret the terms entering the decomposition of $\Delta_{\rm short}$ in (ref) in the context of the examples we introduced in Section (ref). For example, in the farming example where $Y$ is crop yield, $D$ is an indicator of the use of fertilizer, and $A$ is, for simplicity, an indicator of high water usage. In this setting, $\Delta_{\rm dce}^{\rm s}$ captures the average direct causal effect of using fertilizer on the crop yield, where the effect weights units with high and low water usage according to the respective probabilities of these actions happening for the treated, $\pi_1(a)$. The term $\Delta_{\rm ind}^{\rm s}$, in turn, captures a piece of the causal effect of water usage on crop yield that depends on the magnitude of differential water usage between the treated and the untreated. If water usage causally improves crop yield in the absence of fertilizer, and getting an exogenous fertilizer incentivizes units to increase their water usage, this term would be positive.
The following theorem summarizes our discussion above.
Assumption (ref) suffices to interpret a difference-in-means as the total (average) treatment effect $E[Y(1) - Y(0)]$, but it is generally insufficient to remove selection effects–see Example (ref). Moreover, as emphasized in the empirical mediation literature, direct and indirect effects cannot be separately identified in randomized experiments without additional assumptions; see robins1992identifiability. Similar recognition has emerged in applied work, particularly in development economics---for example, Mel/etal:2009, Duflo/etal:2011, and Beaman/etal:2013. For these reasons, we prioritize results derived under Assumption (ref), which enables clean interpretations of regression-based estimands in terms of direct and indirect causal effects.
A seemingly natural, and certainly popular, way to mitigate the presence of indirect effects and obtain an estimand that satisfies strong sign preservation is to control for the other actions linearly as in (ref); an approach we called the long regression. Our main result below shows that the slope coefficient $\Delta_{\rm long}$ in (ref) admits a decomposition similar to that derived by $\Delta_{\rm short}$, and thus includes a combination of direct effects and indirect effects. However, except in some special cases, the coefficients multiplying each average partial causal effect, as in Definition (ref), could be negative and so $\Delta_{\rm long}$ may fail to satisfy strong sign preservation even in the absence of indirect effects. We formalize this below and provide a proof in Appendix (ref).
Theorem (ref) shows that $\Delta_{\rm long}$ can be decomposed into direct and indirect effects, but it leaves open the possibility that the coefficients entering each of these terms could, in general, be negative. An immediate implication is that, except in the special case where the actions in $A$ are all mutually exclusive binary variables, which includes the case where $A$ is a scalar binary variable as a special case, the term $\Delta_{\rm dce}^{\rm l} $ could be negative even if $\mu(1,a)-\mu(0,a)>0$ for all $a\in\mathcal A$. This is because $\omega _{\rm dce}(a)$ may be negative for some $a\in \mathcal{A}$. As a result, $\Delta_{\rm long}$ generally does not satisfy strong sign preservation for the following two reasons. First, it may be possible that $\Delta_{\rm ind}^{\rm l}<-\Delta_{\rm dce}^{\rm l}$, so that the indirect effect dominates the direct effect. This phenomenon is the same as the one we discussed for the short regression. Second, even in the absence of indirect effects, where $\Delta_{\rm ind}^{\rm l}=0$, the term $\Delta_{\rm dce}^{\rm l}$ could be negative by itself even if $\mu(1,a)-\mu(0,a)>0$ for all $a\in\mathcal A$, due to $\omega _{\rm dce}(a)<0$ for some $a\in\mathcal A$. This second possibility represents a stark distinction between the long regression estimand, $\Delta_{\rm long}$, and the short regression estimand, $\Delta_{\rm short}$, in the sense that $\Delta_{\rm long}$ does not even measure a total causal effect of $D$ on $Y$ without additional assumptions, cf.\ Remark (ref). It is also important to emphasize that this result does not depend on the distribution of $Y$ given $(A, D)$. As a consequence, both $\Delta_{\rm long}$ and $\Delta_{\rm dce}^{\rm l}$ can be arbitrarily negative—even if $\mu(1,a) - \mu(0,a) > 0$ for all $a \in \mathcal{A}$—when the actions are not mutually exclusive and binary.
We emphasize that the results in Theorem (ref) are all equivalent. Obtaining an easy-to-interpret characterization of $\Delta_{\rm long}$ in terms of partial effects when the actions are all binary and mutually exclusive should not come as particularly surprising given related results on the properties of this estimand in different but related settings zhao/ding:22,goldsmithpinkham/etal:22. To the best of our knowledge, however, a novel lesson of Theorem (ref) is that this is precisely the only way to guarantee a proper weighted average representation for $\Delta_{\rm long}$ for any distribution of $( A,D)$. This result could also be interpreted in the context of “bad controls” discussed in angrist2008mostly, as it clarifies the unique way in which seemingly bad controls could actually become “good controls”.
The possibility of \( \omega^{\rm l}_{\rm dce}(a) \) being negative for some \( a \in \mathcal A \) does not depend on pathological data-generating processes. It can occur in relatively simple settings with reasonable distributions for \( (A,D) \). This raises a significant concern for the use of linear-in-\( A \) regressions, as they can lead to results that are difficult to interpret and, in many cases, offer no improvement over the short regression in (ref). Below we illustrate this situation with two canonical simple cases: one where $A$ is a scalar random variable taking multiple values, and another one where $A = (A_1,A_2)$ with $A_1$ and $A_2$ being binary.
Consider first the case where $A=A_1$ is a scalar random variable taking values in $\mathcal A_1 = \{0,1,2,\dots,\bar{\rm a}_1\}$. The regression in (ref) simplifies to
Theorem (ref) provides general closed-form expressions for $\{\omega^{\rm l}_{\rm dce}(a):a\in\mathcal A\}$ and $\{\omega^{\rm l}_{\rm ind}(a):a\in\mathcal A\}$ that, when applied to this specific example, lead to
where $p=P\{D=1\}$. From this expression, it follows that any distribution of $(A,D)$ for which $\operatorname*{Cov}(D,A_1)(a-E(A_1))>\operatorname*{Var}(A_1)(1-p)\pi_1(a)$, would lead to negative weights. For example, consider the case where $p=0.8$, $\bar{\rm a}_1=3$, $\{ A|D=1\} \sim {\rm Bi}(3,0.8) $, and $\{ A|D=0\} \sim {\rm Bi}(3,0.2) $, where ${\rm Bi}(n,\pi)$ denotes a Binomial distribution with $n$ trials and success probability $\pi$. In this case, $\omega^{\rm l}_{\rm dce}(3)=-0.41<0$. Figure (ref) plots the weights $\omega_{\rm dce}^{\rm l}(a)$ as a function of $p$ and shows that $\omega^{\rm l}_{\rm dce}(3)$ is negative for any $p>0.4$ in this example.
Next, consider the case where $A=(A_1,A_2)$ with $A_1$ and $A_2$ both being binary variables, so that $\mathcal A = \{(0,0),(1,0),(0,1),(1,1)\}$. The regression in (ref) simplifies to
The closed-form expressions for $\{\omega^{\rm l}_{\rm dce}(a):a\in\mathcal A\}$ and $\{\omega^{\rm l}_{\rm ind}(a):a\in\mathcal A\}$ derived in Theorem (ref) also simplify to this case and lead to simple conditions for which
That is, whenever one of the average partial causal effects gets a positive weight, the other necessarily gets a negative one. As an illustrative example, when $\operatorname*{Cov}(A_1,A_2)=0$,
and
Using the expressions in Theorem (ref), we obtain $\omega^{\rm l}_{\rm dce}(1,0)=-\omega^{\rm l}_{\rm dce}(1,1)=-0.30$, which one more time illustrates that negative weights arise naturally in settings with non-pathological DGPs. In the proof of Theorem (ref), we present even simpler counter-examples that also illustrate how the weights $\{\omega^{\rm l}_{\rm ind}(a):a\in\mathcal A\}$ are generally non-zero and potentially negative, as well as how $\omega^{\rm l}_{\rm dce}(a)$ may be negative without necessarily satisfying (ref).
The results in Theorem (ref) are, to the best of our knowledge, novel—though they are certainly related in spirit to a large literature in biostatistics and epidemiology concerned with mediators and indirect effects. Given the breadth of that literature, we highlight a few representative contributions rather than attempting an exhaustive review.\footnote{Two illustrative examples that were suggested by a referee are groenwold/etal:2021 and wilcox/weinberg/basso:2011. The first paper presents a decompositions of certain regression estimands similar in form to ours, but do so under the assumption that all variables are linearly related to one another—an assumption that, as we show below, has strong implications for interpretation. The second paper warns against including certain variables (such as gestational age) as controls in regression analyses, noting the potential for collider bias. Their critique is motivated by a specific empirical context and supported by simulation, but they do not offer a formal decomposition nor do they analyze the unadjusted case (analogous to our short regression).} While our results are thematically aligned with several ideas in this literature, we are not aware of prior work that establishes the same characterizations with the level of formality and generality presented here. A key contribution of our paper is to show that the case in which the components of $A$ are binary and mutually exclusive is essentially the only one in which the long regression can be interpreted as capturing direct causal effects—without requiring additional assumptions such as linearity. This insight, which is formalized in Theorem (ref), sharpens and unifies a number of informal arguments that appear across distinct literatures.
Perhaps the most relevant work for our purposes is imai2010, who study the interpretation of the long regression—originally popularized by baron1986moderator—under a linear potential outcome model and an assumption they call sequential ignorability. We restate that assumption below to facilitate comparison.
The results in imai2010 about the long regression in (ref) invoke (a) sequential ignorability, (b) a scalar random variable $A$ (though not necessarily binary), and (c) a linear model for $\mu(d,a)$ in $(d,a)$. Under these conditions (a)-(c), and ignoring the $X$ for simplicity, imai2010 shows that $\Delta_{\rm long}$ identifies $\bar\zeta=\bar\zeta(1)=\bar\zeta(0)$ where $\bar\zeta(d) \equiv E[Y(1,A(d))] - E[Y(0,A(d))] = \sum_{a\in \mathcal A}(\mu(1,a)-\mu(0,a))\pi_{d}(a)$, and the equality follows from Assumption (ref) implying $ Y(d',a)\perp A(d)$; see Lemma (ref) in Appendix (ref). The linear model for $\mu(d,a)$ implies that the difference $\mu(1,a)-\mu(0,a)$ does not depend on the value of $a$, and so it is just a constant that we can denote by $\bar \zeta$ without loss of generality. The fact that $\bar\zeta=\bar\zeta(1)=\bar\zeta(0)$ then follows from $\sum_{a\in \mathcal A}\pi_{d}(a)=1$ for $d\in \{0,1\}$.
The additional assumptions (a)-(c) mentioned above have implications on the conclusions of Theorem (ref), which does not invoke any of these assumptions. In particular, the linearity of $\mu(d,a)$ implies that $\Delta^{\rm l}_{\rm dce}$ in (ref) equals $\bar \zeta \sum_{a\in\mathcal A}\omega^{\rm l}_{\rm dce}(a)=\bar \zeta$ by the weights adding up to one according to Theorem (ref). The same linearity assumption also implies that $$ \Delta_{\rm ind}^{\rm l} \equiv \sum_{a\in \mathcal{A}}\omega^{\rm l}_{\rm ind}(a)(\mu (0,a)-\mu (0,0)) \propto \sum_{a\in \mathcal{A}}\omega^{\rm l}_{\rm ind}(a)a = 0~,$$ where the last equality follows from Theorem (ref) in the appendix. We conclude that Theorem (ref) coincides with the results in imai2010 in delivering $\Delta_{\rm long}$ being equal to $\bar\zeta$ under the additional assumption that $\mu(d,a)$ is linear in $(d,a)$. This means that, while sequential ignorability is a stronger assumption than Assumption (ref) (we prove this claim in Lemma (ref) in Appendix (ref)), the main driving force of this result is the linear model for the potential outcomes or, equivalently, the linear model for $\mu(d,a)$.\footnote{Sequential ignorability without a linear model has also been used in the literature on causal mediation analysis to construct semi-parametrically efficient estimators; see tchetgen/etal:12.} As we discussed in Remark (ref), this linearity assumption has been used in economic applications, e.g., heckman2013understanding,fagereng2021wealthy, and while it imposes enough restrictions to provide a clean interpretation to the coefficient $\Delta_{\rm long}$, our results show that such a clean interpretation generally breaks down when $\mu(d,a)$ is not linear in $(d,a)$.
A common variant of the long regression additionally includes interactions between the \(K\) actions, \(A_{1}, \dots, A_{K}\), and the treatment \(D\); that is, the slope coefficient \(\Delta_{\rm inter}\) in (ref). We refer to this as the long regression with interactions. Our result below shows that the slope coefficient \(\Delta_{\rm inter}\) in (ref) admits a decomposition with the same shortcomings as the one we derived for \(\Delta_{\rm long}\), including the possibility of \(\Delta_{\rm inter}\) not satisfying strong sign preservation, even in the absence of indirect effects. We formalize this below and provide a proof in Appendix (ref).
Theorem (ref) is analogous to Theorem (ref) and has very similar implications. Except in the special case where the actions in $A$ are all mutually exclusive binary variables, which includes the case where $A$ is a scalar binary variable as a special case, the term $\Delta_{\rm dce}^{\rm i} $ could be negative even if $\mu(1,a)-\mu(0,a)>0$ for all $a\in\mathcal A$. This is because $\omega^{\rm i}_{\rm dce}(a)$ may be negative for some $a\in \mathcal{A}$.\footnote{The possibility of \( \omega^{\rm i}_{\rm dce}(a) \) being negative for some \( a \in \mathcal A \) can occur in simple settings with reasonable distributions for \( (A,D) \); see the proof of Theorem (ref).} As a result, $\Delta_{\rm inter}$ generally does not satisfy strong sign preservation for the same two reasons that $\Delta_{\rm long}$ fails to do so. First, it is possible that $\Delta_{\rm ind}^{\rm i} < -\Delta_{\rm dce}^{\rm i}$, and second, even if $\Delta_{\rm ind}^{\rm i} = 0$, the term $\Delta_{\rm dce}^{\rm i}$ itself can be negative—despite $\mu(1,a) - \mu(0,a) > 0$ for all $a \in \mathcal{A}$—due to negative weights. Again, this second failure mode distinguishes $\Delta_{\rm long}$ and $\Delta_{\rm inter}$ from $\Delta_{\rm short}$.
While Theorem (ref) focuses on the properties of the estimand $\Delta_{\rm inter}$ in settings with interactions terms, it is most often the case that the analyst would rather focus on the estimand $\Delta_{\rm inter} + \sum_{j=1}^K \lambda_j E[A_j] $ (or, simply, $\Delta_{\rm inter} + \sum_{j=1}^K \lambda_j a_j $ for given values $a_j$, $j=1,\dots,K$). In Lemma (ref) in Appendix (ref), we show that
where the “weights” $\{(\omega_{\mathrm{dce}}^{\mathrm{i}\star}(a),\omega_{\mathrm{ind}}^{\mathrm{i}\star}(a)):a\in\mathcal A\}$ may be negative in general, and thus leading to an estimand with similar properties to those of $\Delta_{\rm inter}$. The one special case where the estimand $\Delta_{\rm inter} + \sum_{j=1}^K \lambda_j E[A_j] $ identifies the intended causal effects is when $\mu(d,a)$ is assumed to take the functional form $\mu(d,a) = \kappa_0 + \kappa_1 d + \sum_{j=1}^K \kappa_{2,a}a_j + d\sum_{j=1}^K \kappa_{3,j} a_j$, which is equivalent to assuming that the conditional mean of the observed outcome, $Y$, is correctly specified in the interaction regression in (ref). Lemma (ref) in Appendix (ref) shows that $\Delta_{\rm inter} + \sum_{j=1}^K \lambda_j a_j = \mu(1,a) - \mu(0,a)$ in this case, delivering an average partial causal effect given $a=(a_1,\dots,a_K)$, as defined in Definition (ref). It follows from these results that a clean interpretation of $\Delta_{\rm inter}$ or $\Delta_{\rm inter} + \sum_{j=1}^K \lambda_j a_j$ in terms of the definitions introduced in Section (ref) essentially depends on a correctly specified parametric model for potential outcomes and does not generally apply to nonlinear models, similarly to our results about the long regression in Section (ref).
Theorems (ref) and (ref) show that adding other actions linearly in the regression is generally useful only when the actions are binary and mutually exclusive. If we could make the actions binary and mutually exclusive, we would obtain an estimand free from indirect effects and with strong sign preservation. This is possible by considering the slope coefficient \(\Delta_{\rm sfe}\) in (ref), where the regression controls for all possible values of \(A\), i.e., \(\{I\{A=a\}:a\in\mathcal A\}\). We call this a strata-fixed effects regression, due to its connection to the use of strata fixed effects in randomized controlled trials with covariate adaptive randomization; see bugni/canay/shaikh:2018,bugni/canay/shaikh:2019. Our result below shows that \(\Delta_{\rm sfe}\) in (ref) has a decomposition free from indirect effects (see Appendix (ref) for a proof).
Theorem (ref) shows that $\Delta_{\rm sfe}$ identifies an average direct causal effect as in Definition (ref). Importantly, it does not contain indirect effects and, as a result, $\Delta_{\rm sfe}$ satisfies strong sign preservation as in Definition (ref). The weight $\omega_{\rm sfe}(a)$ admits a simple representation and depends only on the conditional probabilities that the action $a$ happens for the treated and control group, $\pi_{1}(a)$ and $\pi_{0}(a)$. These weights are generally different than the weights associated with the direct effect in the short regression, $\Delta_{\rm dce}^{\rm s}$, which are simply $\pi_{1}(a)$, unless $D$ and $A$ are independent. We emphasize that the result in Theorem (ref) does not require the other actions, $A$, to be singled-valued or mutually exclusive.
The estimand $\Delta_{\rm sfe}$ has been studied in other settings. For example, angrist:98 studies regressions of $Y$ on $D$ under Assumption (ref) and considers a linear regression that saturates on the covariates, $X$, to obtain an expression that parallels the one in Theorem (ref); see angrist:98. Indeed, Assumption (ref) implies that $Y(d,a)\perp D|A$ which makes the connection to angrist:98 immediate. In this sense, our result illustrates another instance where stratifying confounding variables leads to easy-to-interpret results (though, it is important to notice that this result relies on the fact that the treatment variable $D$ in our case is binary, c.f. goldsmithpinkham/etal:22). As another example, bugni/canay/shaikh:2018,bugni/canay/shaikh:2019 study the properties of this estimand and present results on how to properly compute standard errors in randomized controlled experiments with covariate adaptive randomization. These papers, however, do not represent $\Delta_{\rm sfe}$ as a weighted average of causal effects since the strata are not viewed as counter-factual features in such experiments.
We now turn our attention to the last set of estimands we study in this paper; the slope coefficient $\Delta_{\rm sat}$ in (ref). As we have stated in the introduction, under Assumption (ref), it follows that $\mu(d,a)$ is immediately identified from $E[Y|D=d,A=a]$ for any $d\in \{0,1\}$ and $a\in\mathcal A$ and so identification of any contrast of means $\mu(d,a)$ is straightforward. From this, it immediately follows that the same result could be achieved by running a saturated regression, as in (ref), that we re-write here for readability, $$ Y = \sum_{a\in\mathcal A} \gamma(a)I\{A=a\}+ \sum_{a\in\mathcal A} \Delta_{\rm sat}(a)I\{A=a\} D + \epsilon~.$$ Under Assumption (ref), standard results on saturated regressions imply that $\Delta_{\rm sat}(a)=\mu(1,a)-\mu(0,a)$ for all $a\in\mathcal A$, and so $\Delta_{\rm sat}(a)$ captures an average causal partial effect of $D$ on $Y$ for each value of the other actions, $a\in\mathcal A$, aligned with Definition (ref). For completeness, we state and prove this result formally in Theorem (ref) in the appendix. The same result also shows that replacing Assumption (ref) with Assumption (ref) leads to a decomposition of $\Delta_{\rm sat}(a)$ that includes a selection term, as it was also the case for the other estimands we considered. Finally, we note that this approach is equivalent to a multi-way ANOVA, since the mean for the outcome variable depends exclusively on categorical variables.
In this paper we analyze settings where the researcher seeks to estimate the average direct causal effect of a binary treatment \( D \) on an outcome \( Y \), in situations where the outcome is realized with a delay and other endogenous actions \( A \) may occur in the interim. A central premise of our analysis is that researchers often use regression-based estimands with the intent of isolating a direct causal effect of \( D \) on \( Y \), not a total effect that also includes indirect pathways through \( A \).
We focus on several popular estimands arising from regressions that control for \( A \) in various ways—linearly, with interactions, or not at all—and decompose them to assess whether they admit a causal interpretation. Our decompositions are algebraic and do not rely on identification assumptions, but we examine the additional conditions under which causal interpretations are possible. The results are “negative" in spirit: they show that, even under strong exogeneity assumptions, these commonly used regression-based estimands generally do not recover the average direct causal effect of interest unless we impose restrictive assumptions (e.g., linearity in the potential outcomes model). We highlight that only in the special case where the actions are binary and mutually exclusive does the long or interacted regression estimand admit a causal interpretation of a direct effect without further assumptions—and our result shows that this is an if and only if condition.
The goal of our exercise is not to propose new estimators or to discuss nonparametric identification of direct and indirect effects, but rather to clarify the conditions under which standard approaches yield meaningful causal parameters. We hope this work helps practitioners recognize the limitations of common estimands and adopt a more deliberate approach when interpreting them as capturing direct effects.
\addcontentsline{toc}{section}{References}
\setcounter{lemma}{0} \setcounter{theorem}{0} \setcounter{corollary}{0} \setcounter{equation}{0} \setcounter{remark}{0}