Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
89,163 characters · 20 sections · 62 citation commands
Triple Difference Designs with Heterogeneous Treatment Effects
\setcounter{section}{0}
Triple difference designs (also known as triple difference-in-difference or difference-in-difference-in-difference designs) are, increasingly, a popular research design for estimating causal effects. Triple difference (3D) designs rely on comparisons across three dimensions, for example, across treatment assignment, time, and another characteristic of interest. The simplest design takes on a $2 \times 2 \times 2$ form, with binary variation in each of the three dimensions. Triple difference designs allow researchers to identify causal effects in cases when comparison across only one dimension (for example, a pre-post comparison) or two dimensions (for example, a difference-in-difference design comparing trends over time between treatment and control group) are confounded. In particular, triple difference designs are often useful when the parallel trends assumption required for a difference-in-difference design is not satisfied.
Although the use of triple difference designs has been increasing in recent years oldenTripleDifferenceEstimator2022, their properties are still little-studied. Prior work has not fully addressed how underlying heterogeneity in treatment effects can affect estimation and interpretation in a triple difference design. A recent review of the literature on difference-in-difference methods has pointed out that further study of triple difference methods and guidance for researchers is needed rothWhatsTrendingDifferenceindifferences2023. This paper attempts to fill this gap by offering a formal discussion of triple difference designs and estimators under treatment effect heterogeneity.
First, I discuss the interpretation of the triple difference parameters of interest and clarify some ambiguity in the literature about the assumptions necessary for a triple difference design. I consider a setup with the three sources of variation being time, treatment assignment, and subgroup. For example, this setup would apply to the analysis of a policy implemented at the state level, which is thought to affect one group of individuals, such as married people, more strongly than others. Previous work, such as oldenTripleDifferenceEstimator2022,ortiz-villavicencioBetterUnderstandingTriple2025, has shown that a triple difference design can identify the average treatment effect on the treated (ATT) for the subgroup of interest (eg, married people) by assuming that the comparison subgroup (eg, unmarried people) is unaffected by the treatment. However, in practice, researchers may not be willing to make this assumption and instead are interested in estimating a different parameter: the difference in ATTs between the subgroup of interest and the comparison subgroup, which I call the DATT. I distinguish these parameters and compare the assumptions required to identify them.
The distinction between the ATT and DATT is important in many triple difference applications. In some designs, there is no assumption that any subgroup is unaffected by the treatment and, instead, the DATT capturing the relative effect of the treatment between the subgroups is the primary parameter of interest. For example, derenoncourtMinimumWagesRacial2020 use a triple difference strategy to study whether an increase in the minimum wage had a larger impact in the US South relative to other regions. In other cases, the unaffected subgroup assumption is made but may need further evidence to support it. For example, gruberIncidenceMandatedMaternity1994 compares the impacts of legislation requiring insurance coverage for childbirth costs on married men and women of childbearing age, compared to single men and those older than childbearing age. The comparison subgroup would not be unaffected if there are spillovers, such as employers substituting between groups of workers. A similar point is made in baumEffectStateMaternity2003. When there is doubt about whether one subgroup is truly unaffected by the treatment, the DATT can still be identified.
Next, I consider the interpretation of the DATT when there is heterogeneity in treatment effects. I show that, when observations' treatment effects (their sensitivities to the treatment) are correlated with subgroup status, the DATT does not represent the causal effect of belonging to the subgroup of interest relative to the comparison subgroup. For example, this case arises when members of one subgroup would have been more sensitive to the treatment than members of the comparison subgroup even if they had belonged to the comparison subgroup.
To identify the causal effect of subgroup status, I propose a new parameter of interest, the causal DATT or CDATT. I discuss both the two-period design and the staggered treatment design, addressing the concerns raised in strezhnevDecomposingTripleDifferencesRegression2023 and in line with the difference-in-differences literature (eg, rothWhatsTrendingDifferenceindifferences2023). I introduce and discuss the assumptions necessary to identify the CDATT and compare with the DATT.
In many applications, the CDATT is as much of interest as the DATT. For example, gruberIncidenceMandatedMaternity1994 studies whether, following mandates for insurance coverage of childbirth costs, workers' wages fell according to their valuation of the coverage. That is, the author is interested in differences between the subgroups caused by likelihood to use the benefits (proxied by demographics, eg, being of childbearing age) rather than, for example, differences caused by differences in occupations between the groups. Similarly, matsaFemaleStyleCorporate2013b use a triple difference strategy to study the impacts of gender quotas for corporate boards in Norway, comparing impacts on publicly listed companies relative to unlisted companies who were exempt. An interesting question is whether the differential impacts of the quota on listed companies are caused by being listed (and, therefore, the regulation) or other differences in firm characteristics, such as firm size, between the subgroups.
I propose estimators for the CDATT and derive their asymptotic properties. When heterogeneity in underlying treatment effects can be modeled using observable characteristics, the CDATT can be identified using an inverse propensity score weighting (IPW) estimator, regression adjustment (RA), or a doubly-robust estimator combining both of these. After deriving the semiparametric efficiency bounds for the CDATT, I show that, when all working models are correctly specified, the doubly-robust estimator is asymptotically efficient.
Next, I present a Monte Carlo simulation study calibrated to the data and approach in gruberIncidenceMandatedMaternity1994 to show the finite-sample properties of these estimators in a case where the DATT and CDATT differ. When treatment effects are correlated with subgroup status, estimates of the DATT diverge sharply from estimates of the CDATT. I show that the doubly-robust estimator of the CDATT attains its semiparametric efficiency bound when the working models are correctly specified.
Finally, I apply these estimators to re-analyze the data in gruberIncidenceMandatedMaternity1994. This analysis attempts to assess the impacts of a policy requiring insurance to cover the costs of childbirth, thereby increasing the cost to employers of hiring workers likely to use these benefits. The author uses a triple difference design to understand whether these costs can be differentially passed on to targeted workers on the basis of demographics. Although analysis of the DATT would seem to suggest that such cost-shifting is possible, analysis of the CDATT offers weaker evidence of this. This would suggest that any differential cost-shifting may occur on the basis of other personal or job characteristics, rather than demographic group. Researchers estimating DATT parameters should be cautious to avoid interpreting these as CDATT parameters.
Contributions. This work contributes directly to the literature on triple difference designs (eg, gruberIncidenceMandatedMaternity1994, oldenTripleDifferenceEstimator2022,ortiz-villavicencioBetterUnderstandingTriple2025). I discuss parameters of interest that are identified under more minimal assumptions than those suggested by oldenTripleDifferenceEstimator2022,ortiz-villavicencioBetterUnderstandingTriple2025. In both of those papers, the authors assume that a comparison subgroup is available which is entirely unaffected by the treatment and focus on identification of the ATT for the subgroup of interest. My work highlights this assumption and that, in many cases, this assumption may be not needed or not justified. Instead, the difference in ATTs between the subgroups is the parameter of interest or the only parameter that can be identified. I also discuss the interpretation of this parameter when there is heterogeneity in treatment effects and highlight that, even under the stronger assumptions used by oldenTripleDifferenceEstimator2022,ortiz-villavicencioBetterUnderstandingTriple2025, causal statements about the difference between subgroups cannot be made without additional identification assumptions. To address this, I propose an alternative parameter of interest which makes causal comparisons between subgroups.
This work also contributes to the growing literature on difference-in-differences and related designs under treatment effect heterogeneity (eg, rothWhatsTrendingDifferenceindifferences2023, callawayDifferenceinDifferencesContinuousTreatment2021a, sunEstimatingDynamicTreatment2021, dechaisemartinTwoWayFixedEffects2020, goodman-baconDifferenceindifferencesVariationTreatment2021). It adds to this literature by addressing issues that arise in triple difference designs, as pointed out by strezhnevDecomposingTripleDifferencesRegression2023, and by addressing treatment effect heterogeneity between subgroups of the population of interest. In this way, this paper relates to work which addresses comparisons between “more treated" and “less treated" groups in difference-in-difference designs dechaisemartinFuzzyDifferencesinDifferences2018a,dechaisemartinTwoWayFixedEffects2020,xuFactorialDifferenceinDifferences2025 (“fuzzy” difference-in-differences) and difference-in-differences with continuous treatments callawayDifferenceinDifferencesContinuousTreatment2021a. Although previous work in difference-in-differences has mentioned simple ways of extending difference-in-difference results to a triple difference framework (eg, rothEfficientEstimationStaggered2023,goodman-baconDifferenceinDifferencesVariationTreatment2018), previous work has not addressed the unique forms of treatment effect heterogeneity that can arise in a triple difference design.
Finally, this work also contributes to a literature on estimators and estimation of difference-in-difference parameters rothEfficientEstimationStaggered2023,santannaDoublyRobustDifferenceindifferences2020, callawayDifferenceinDifferencesMultipleTime2021, abadieSemiparametricDifferenceinDifferencesEstimators2005a. I build on this work to describe doubly-robust estimators for triple difference designs and show that the estimator for my proposed parameter of interest is semiparametrically efficient.
I introduce a setup based on a potential outcomes framework imbensCausalInferenceStatistics2015. Treatment status for individual $i$ at time $t = 0, \dots, \mathcal{T}$ is given by $W_{it}$. I consider a binary treatment, $W_{it} \in \{0,1\}$. Let $Y_{it}(0)$ and $Y_{it}(1)$ represent the potential outcomes for individual $i$ at time $t$ under $W_{it} = 0$ and $W_{it} = 1$, respectively, such that $Y_{it} = Y_{it}(1)W_{it} + Y_{it}(0)(1-W_{it})$. Finally, let individuals belong to subgroup $s_i \in \mathcal{S}$. For example, imagine that individuals may be married or not, with $s_i \in \{\text{married}, \text{not married}\}$. This setup also extends naturally to a case with multiple subgroups, such as $s_i \in \{\text{married}$, $\text{never married}$, $\text{previously married}\}$.
Under this assumption, as in callawayDifferenceinDifferencesMultipleTime2021, once a treatment “turns on" at a given time $\tau$, the unit remains treated for all $t \geq \tau$. For example, consider the problem of estimating the impacts of a state-level policy. The assumption is satisfied if, once this policy takes effect in a given state, it remains state law for the future.
The following text will consider individuals to be randomly sampled in a panel data setting. That is, $\{Y_{it}, W_{it}, X_{it}, S_{i}\}_{t \in \mathcal{T}}$ are independently and identically distributed (iid). As such, the $i$ subscript will be suppressed. The repeated cross-section case is discussed in Appendix (ref).
In this section, I introduce and discuss the interpretation of the difference in average treatment effects on the treated ($DATT$) parameter. In the canonical triple difference design, such as that studied by oldenTripleDifferenceEstimator2022, there are two time periods, a binary treatment, and two subgroups. I extend this design in two ways: by allowing for more than two subgroups and by allowing for staggered treatment designs.
In both the binary and staggered treatment designs, I show that the interpretation of these $DATT$ parameters may be affected by treatment effect heterogeneity, even if the above parameters are identified. I then introduce and discuss a new parameter of interest, the causal difference in average treatment effects on the treated.
In the two period case, I rewrite potential outcomes in terms of both potential treatment status and potential subgroup status. That is, $Y_t(W; S)$ represents the potential outcome under (possibly counterfactual) treatment status $W$ and (possibly counterfactual) subgroup $S$. We have $Y_t(W) = \sum_{s\in\mathcal{S}}Y_t(W; s)\mathbbm{1}\{S = s\}$. I also define a potential $ATT$, which I call $ATT(s)$. This represents the $ATT$ if all individuals had belonged to a generic subgroup $s$: $$ATT(s) = \mathbb{E}[Y_{1}(1; s) - Y_1(0;s) | W=1] $$ Finally, define for any subgroups $s$ and $s'$, $[ATT(s)|S=s'] = \mathbb{E}[Y_1(1; s) - Y_{1}(0; s) | W =1, S=s']$.
In the staggered treatment case, define, for any subgroups $s$ and $s'$, the potential outcomes $Y_t(G;S)$ analogously to the above. Then, for a generic $g, t,s$ and $s'$, $$ATT(g,t;s) = \mathbb{E}[Y_{t}(g; s) - Y_{t}(\infty;s) | G=g] $$ $$[ATT(g, t;s)|S=s'] = \mathbb{E}[Y_t(g;s) - Y_{t}(\infty; s) |G=g, S=s']$$
Using these quantities, I decompose the $DATT_{s-s'}$ parameter for the two period case:
and for the staggered case:
The first term, the causal $DATT_{s-s'}$ or $CDATT_{s-s'}$, captures the difference in treatment effects due to subgroup status, among those who (in reality) belonged to subgroup $s$. The second term, which is due to treatment effect heterogeneity, captures how the treatment effects would differ if all individuals had belonged to the same subgroup, with any differences arising from differences in the underlying treatment effects for the individuals that selected into each subgroup. In words, this arises when subgroup status is correlated with treatment effects.
For example, consider a triple difference design analyzing the impacts of a policy change on the employment of married individuals ($s$) and unmarried ($s'$) individuals. Treatment effect heterogeneity will affect the $DATT_{s-s'}$ to the extent that those who actually were married in this sample have different underlying treatment effects than those who actually were not married. For example, suppose that married people tend to be in different occupations than unmarried people, and suppose that individuals in those occupations are more sensitive to a policy change. Then, the average treatment effect on the treated of the married group, had they been unmarried (but still in those occupations), differs from that of the unmarried group. Meanwhile, the true difference in treatment effects caused by marital status would be estimated by the $CDATT_{s-s'}$.
Four applications below highlight cases in which researchers might be interested in studying the $CDATT_{s-s'}$ and how it might diverge from the $DATT_{s-s'}$.
gruberIncidenceMandatedMaternity1994 uses a triple difference design to evaluate the difference in impacts of a policy requiring insurance companies to provide coverage for childbirth costs. The author compares the effect of being in a group likely to use these benefits (married individuals and single women of childbearing age) relative to groups that are not likely to use these benefits (single men and older adults). The $CDATT_{s-s'}$ would allow the researcher to understand whether differences in impacts between these subgroups are caused by differences in likelihood to use the benefits (proxied by demographics) or by other characteristics correlated with subgroup, such as occupation.
baumEffectStateMaternity2003 uses a similar design to investigate the impacts of maternity leave benefits on women with children and women of childbearing age relative to single men. The $CDATT_{s-s'}$ would allow the author to interpret differences in the impacts of maternity leave benefits for these subgroups as being caused by subgroup status (ie, caused by women being eligible for the benefits) rather than by other characteristics that differ between the subgroups. Understanding whether the differences between subgroups are causal may have important implications for policy.
matsaFemaleStyleCorporate2013b study the impacts of quotas for women's representation on corporate boards in Norway. To do this, they use a triple difference design which compares the effects of the policy on listed (public) companies and unlisted companies, which were exempt from the quota. Showing that the quota's impact on the listed companies was caused by the regulation requires a causal statement about the comparison between subgroups. That is, the researcher might be interested to understand whether listed companies were impacted because they were listed (and, therefore, bound by the regulation), or because of another characteristic that differs between listed and unlisted companies, such as firm size.
In another application of a triple difference design, derenoncourtMinimumWagesRacial2020 investigate whether the impacts of the minimum wages imposed by the Fair Labor Standards Act were larger in the US South compared to other regions. The authors' motivation for studying this is to unpack the impact of this policy on racial inequality, and so an interesting investigation would be whether differential impacts are caused by being in the South (eg, due to regional histories) or by differences in other characteristics correlated with region (eg, worker or firm characteristics).
The treatment effect heterogeneity arising between subgroups may be particularly concerning in cases where the policy studied can cause selection into the subgroups analyzed based on treatment effects. For example, if a policy intervention causes those who are more sensitive to its impacts to get married or have children, then those subgroups will disproportionately contain individuals who are more sensitive to the treatment. A similar concern in the difference-in-difference setting is discussed by blundellAlternativeApproachesEvaluation2009, who consider the challenge of estimating the returns to education when individuals select into education based on the returns they expect to receive.
Finally, a common assumption behind triple difference designs, discussed more formally below, involves treating one subgroup as a “control subgroup", which is assumed to be unaffected by a given policy change. Even under this assumption, a causal interpretation of the differences between subgroups requires an additional assumption on the potential outcomes of the subgroup of interest.
In this section, I show that identification of the $DATT$ requires an assumption limiting anticipation of the treatment and a parallel-trends-type assumption. I point out that additional assumptions on the comparison subgroup can allow researchers to recover the $ATT$ in a triple difference design. Finally, I highlight that identification of the $CDATT$ requires an additional assumption on treatment effect heterogeneity. The proofs of these propositions appear in Appendix (ref). From here, through the rest of the paper, results will be derived for the staggered treatment case. The two-period case can be thought of as a special case of this.
The following assumptions are used to identify the parameters of interest.
This assumption is the same as in callawayDifferenceinDifferencesMultipleTime2021 and ensures that untreated potential outcomes are observed for all units in the pre-treatment periods.
Assumptions (ref) and (ref) compare closely with the difference-in-difference parallel trends assumption, and are equivalent to a parallel trends assumption on the gap in outcomes between subgroups. These assumptions require that the trends in the gap in outcomes between subgroup $s$ and $s'$ be parallel between treated and not-yet-treated or never-treated units. These are an extension of the identification assumptions outlined by callawayDifferenceinDifferencesMultipleTime2021 for difference-in-differences with staggered treatment. In the application to identifying the impacts of a policy on married relative to unmarried individuals, this amounts to assuming that, had the policy not been implemented, the difference in outcomes between married and unmarried individuals would have followed the same trend in states that were treated at time $g$ and states that were not-yet-treated at that time (or never treated). This means assuming that there are no other variables that both are correlated with how early or late a state adopted its policy and the trend in the gap in outcomes between married and unmarried individuals.
This result extends the identification results in oldenTripleDifferenceEstimator2022 in three ways. First, it extends to the staggered treatment case using the methods by callawayDifferenceinDifferencesMultipleTime2021. Second, it allows for the possibility that members of subgroup $s'$ with $G=g$ may be affected by the treatment, while oldenTripleDifferenceEstimator2022 assume that this subgroup is not affected by the treatment by assuming that we observe untreated potential outcomes for those with $S=s'$ and $G=g$. The assumption of an unaffected subgroup will be discussed in further detail below. Third, this result highlights that, in order to make comparisons between multiple subgroups, multiple parallel gaps assumptions are needed. The proof appears in Appendix (ref) (Proof (ref) and (ref)). \\
Remark. In both cases above, the researcher may make comparisons between more than two subgroups, for $s \in \mathcal{S}$. This does not require mutually exclusive subgroups; however, researchers should note that identifying variation will come from the difference in subgroups, so that comparisons between groups that are too similar may lack power. For more discussion of difference-in-difference with fuzzy treatments, see dechaisemartinFuzzyDifferencesinDifferences2018a,dechaisemartinTwoWayFixedEffects2020,galindo-silvaFuzzyDifferenceinDiscontinuitiesIdentification2021b, xuFactorialDifferenceinDifferences2025. A full treatment of triple difference with a continuous subgroup variable is outside the scope of this paper. Difference-in-differences with a continuous treatment is discussed in callawayDifferenceinDifferencesContinuousTreatment2021a. \\
As introduced above, neither Assumption (ref) nor Assumption (ref) is enough to identify the $CDATT$ when treatment effects are heterogeneous. From here, I turn to several assumptions on treatment effect heterogeneity.
As described above, the triple difference $DATT_{s-s'}$ is equivalent to the difference in the $ATT$ for subgroup $s$ and the $ATT$ for subgroup $s'$. However, individually, neither the $ATT$ for subgroup $s$ nor the $ATT$ for subgroup $s'$ can be identified under the parallel gaps assumption.
To recover the $ATT$ using a triple difference design, we must impose at least one assumption on the treated counterfactual outcomes. One common approach would be to assume that one subgroup was unaffected by the treatment. For example, to recover the impact of a policy on married individuals, we might assume that unmarried individuals would be unaffected by the policy.
Under Assumption (ref) for subgroup $s'$, $DATT_{s-s'} = [ATT | S=s] - [ATT | S=s'] = [ATT |S=s]$. The population $ATT$ can be recovered by averaging $[ATT|S=s]$ and $[ATT|S=s'] = 0$ according to the shares of each subgroup in the population.
This assumption is made implicitly by oldenTripleDifferenceEstimator2022, as they assume that $Y_t(\infty)$ is observed for units with $W_t = 1$ and $S=s'$, that is, they assume that units in $s'$ are unaffected by $W_t$.
An important violation of the unaffected subgroup assumption arises in many practical applications when there is the possibility of spillovers between subgroups. For example, a researcher using a design like that in gruberIncidenceMandatedMaternity1994 might assume that unmarried men are unaffected by a policy requiring insurance coverage of the costs of childbirth. However, this will not be true if employers substitute between the groups of workers.
Although the unaffected subgroup assumption is not directly testable, some authors have attempted to assess evidence of spillovers between subgroups eligible and ineligible for a program. For example, desiereHowEffectiveAre2022 examine the impacts of a hiring subsidy on job outcomes by comparing a subgroup of older workers (age 45-48) who were eligible for the subsidy and a subgroup of younger workers who were not (age 40-43). The unaffected subgroup assumption will be violated if employers substitute between the older and younger workers. The authors test for evidence of this by comparing impacts on these subgroups with a subgroup of even younger workers. If employers substitute between workers around the eligibility cutoff, then the just-ineligible workers would experience different effects than the even-younger workers.
Alternatively, if Assumption (ref) is not satisfied, the $ATT$ for subgroup $s$ can be bounded under an assumption limiting the magnitude of the $ATT$ for subgroup $s'$. For example, this might be relevant in cases where it is believed that the comparison subgroup is not unaffected but is only affected a small amount.
In this section, I describe two identification assumptions that can be used, along with the assumptions above, to identify the $CDATT$.
Assumptions (ref) and (ref) limit the treatment effect heterogeneity between the subgroups. Assumption (ref) implies that underlying average treatment effects on the treated would have been the same for members of subgroup $s$ and $s'$, had they all belonged to subgroup $s'$. In a sense, this assumption says that, had the identities of those in each subgroup been swapped (that is, had those in subgroup $s$ actually belonged to $s'$), the average treatment effect on the treated for this subgroup would be unchanged. This assumption is violated whenever the subgroups differ in their sensitivity to the treatment.
Assumption (ref) weakens this assumption by requiring that there is no treatment effect heterogeneity between the groups after conditioning on control variables $X$. Assumption (ref) is a more general form of Assumption (ref) if $X$ is allowed to be degenerate.
For example, in an application comparing the outcomes of married women with those of single men, suppose that married women tend to be more sensitive to a labor market shock because they tend to work in more sensitive occupations. In this case, the heterogeneity between the groups can be described in terms of heterogeneity in job characteristics, satisfying Assumption (ref) by taking $X$ to be occupation. After conditioning on occupation, there is no difference in sensitivity between the groups. \\
Remark. A special case of Assumption (ref) is one in which the researcher assumes an average treatment effect on the treated of zero for both subgroups, had they been in subgroup $s'$. For subgroup $s$, this is a statement about counterfactual treatment effects, since they were not observed in subgroup $s'$. In this way, this is stronger than the unaffected subgroup assumption discussed above (Assumption (ref)), which only makes an assumption on the treatment effects of those in subgroup $s'$. In the example where the subgroups are married and unmarried individuals, Assumption (ref) requires assuming that the average treatment effect on the treated for unmarried individuals is zero. Adding Assumption (ref) requires the assumption that, had the subgroup assignments been swapped so that those were actually married were not married, their average treatment effect on the treated would also have been zero. While Assumption (ref) relates only to the potential outcomes for the comparison subgroup, Assumptions (ref) and (ref) also relate to the potential outcomes for the subgroup of interest.
This proposition implies that, when there is no treatment effect heterogeneity, the $DATT_{s-s'}$ and the $CDATT_{s-s'}$ are equivalent. In effect, Assumption (ref) affects the interpretation of the parameters of interest, but not their estimation. The proof appears in Appendix (ref) (Proofs (ref) and (ref)).
To discuss the identification of the $CDATT$ when treatment effect heterogeneity between the subgroups is captured by observable characteristics, I introduce additional notation. Let $S_s = 1$ if $S =s$ and 0 otherwise. Let $G_g = 1$ if $G=g$ and 0 otherwise. Finally, let $C_{ny} = 1$ if $W_t=0, G\neq g$ and 0 otherwise, and let $C_{nev} = 1$ if $G = \infty$ and 0 otherwise.
Now, define propensity scores
and outcome functions
Then, define the following parameters, for $c \in \{ny, nev\}$ : {
where
}
This setup is general in the sense that $X$ may be degenerate.
To ensure that the quantities above are well-defined, one more assumption is needed.
Assumption (ref) ensures a positive probability of belonging to the treated group and subgroup of interest $s$. It also ensures, given covariates $X$, a nonzero probability of belonging to each subgroup-treatment group pair. This assumption extends a similar assumption in callawayDifferenceinDifferencesMultipleTime2021 to the triple difference case.
The proof appears in Appendix (ref) (Proofs (ref), (ref), and (ref)).
If neither Assumption (ref) nor Assumption (ref) holds, it may not be possible to recover the $CDATT$. However, making more limited assumptions on treatment effect heterogeneity can allow partial identification of this parameter.
Assumption (ref) amounts to imposing that, on average, individuals select into the subgroup in which they will experience the larger treatment effect. This is related to the monotone treatment selection assumption suggested by manskiMonotoneInstrumentalVariables2000a. For example, this assumption might be expected to hold in the case where individuals who expect that they will receive the most benefits from a policy targeted towards married individuals choose to get married.
This proposition shows that, if selection into subgroups is economically motivated by subgroup treatment effects, then the $CDATT_{s-s'}$ will always be larger than the $DATT_{s-s'}$. Under this assumption, researchers can interpret the $DATT_{s-s'}$ as a lower bound for $CDATT_{s-s'}$.
In this section, I discuss the properties and estimation of $CDATT^{DR, c}_{s-s'}(g,t)$ for $c \in \{ny, nev\}$.
Semiparametric efficiency bounds are given for difference-in-difference parameters under a conditional parallel trends assumption in santannaDoublyRobustDifferenceindifferences2020. The analysis in this paper follows similarly, using the approach suggested by neweySemiparametricEfficiencyBounds1990 and also used by hahnRolePropensityScore1998.
To simplify notation, assume the sample is limited to those either with $S_s = 1$ or $S_{s'} = 1$, so that $S_{s'} = 1-S_s$.\\
Now, define
Following this, define
The proof appears in Appendix (ref) (Proof (ref)).
As we have seen above, the $DATT_{s-s'}(g,t)$ can be thought of as the difference between two difference-in-difference $ATT(g,t)$ parameters, estimation and inference for which is provided in callawayDifferenceinDifferencesMultipleTime2021 and santannaDoublyRobustDifferenceindifferences2020. When conditioning on controls, as callawayDifferenceinDifferencesMultipleTime2021 point out, several methods may be appropriate, including regression adjustment, inverse probability weighting, and doubly robust approaches. \\
Remark. As pointed out by goodman-baconDifferenceindifferencesVariationTreatment2021, it is also possible to obtain an estimate of the triple difference $DATT$ by estimating a difference-in-differences model on treatment cohort-level gaps between subgroup $s$ and $s'$. For example, if $G$ is assigned at the state level, one may collapse the data to the state level and estimate a weighted difference-in-difference model on the gap in average outcomes between subgroup $s$ and $s'$ in each state. However, aggregating the data from the individual to the state level may reduce the precision of the estimates and preclude the inclusion of individual-level controls. \\
Consider parametric models for the propensity scores $\pi_{g,s}, \pi_{g,s'}, \pi_{c,s}$ and $\pi_{c,s'}$ that take the form $e_{a,b}(X; \theta_{a,b})$ for $a \in \{g,c\}$ and $b \in \{s,s'\}$. Consider also parametric models for the outcome functions $\mu_{g,t}^{s'}$ and $\mu_{c,g,t}^{s'}$ of the form $m_{a,t}^{s'}(X; \beta_{a}^{s'})$ for $a \in\{g,(c,g)\}$. These parametric models are estimated by $e_{a,b}(X; \widehat\theta_{a,b})$ and $m_{a,t}^{s'}(X; \widehat\beta_{a}^{s'})$, respectively.
Let $\mathbb{E}_n [X] = \frac{1}{n} \sum_{i=1}^n X_i$. Then, the following estimates $CDATT^{DR,c}_{s-s'}(g,t)$, for $c \in \{ny, nev\}$:
where
Estimation of the $\widehat{CDATT}_{s-s'}^{DR,c}$ requires specifying both the outcome models and the propensity score models. Even when both the treatment and subgroup status are binary, the propensity score models require estimating the probability of belonging to 4 groups (ie, $P(W=1, S=s |X)$, $P(W=1, S=s'|X)$, $P(W=0, S=s|X)$, $P(W=0, s'|X)$). For this, an approach like a multinomial logit or probit estimation would often be suitable.
Next, I discuss the asymptotic properties of $\widehat{CDATT}^{DR,c}_{s-s'}(g,t)$. Derivation of the asymptotic properties depends on a regularity assumption, which is further laid out in Appendix (ref) (Assumption (ref)). This assumption is standard in the literature, eg, santannaDoublyRobustDifferenceindifferences2020 and callawayDifferenceinDifferencesMultipleTime2021. It puts some smoothness restrictions on the form of the working models, which are satisfied by common models such as linear models and multinomial logit.
For ease of notation when considering multiple time periods, let $\widehat{CDATT}^{DR,c}_{t \geq g}$ denote the vector of all $\widehat{CDATT}^{DR,c}_{s-s'}(g,t)$ with $t \geq g$. Define $CDATT_{t\geq g}$ and $\eta_{g \geq t}^{DR, c}$ analogously.
The proof appears in Appendix (ref).
Proposition (ref) shows that, under the given assumptions, the proposed estimator is asymptotically normally distributed. When the working models are correctly specified, the variance of the estimator corresponds to the semiparametric efficiency bound as given in Proposition (ref). When there is misspecification in either the propensity score or the outcome models, the estimator is still asymptotically normal, but with additional estimation terms adding to the variance. This result is similar to that in santannaDoublyRobustDifferenceindifferences2020. They also discuss estimators with inference that is robust to misspecification of the working models, a result which depends on the specification of the working models.
The asymptotic variance of the estimator can be estimated using a straightforward plug-in estimator, that is, by replacing population values with their sample analogues.
In this section, I use the data and approach by gruberIncidenceMandatedMaternity1994 to illustrate the impacts of estimating the DATT and CDATT by the methods I propose. First, I design a realistic Monte Carlo simulation study calibrated to the data and application in that paper to show the properties of my proposed estimators in several cases where the DATT and CDATT diverge. Next, I re-analyze the data to show that, in this analysis of the impacts of mandated insurance coverage of childbirth costs on labor market outcomes, the DATT and the CDATT can offer different conclusions.
gruberIncidenceMandatedMaternity1994 studies the impacts of legislation requiring the cost of childbirth (“maternity benefits") to be covered by employers' health insurance policies. The author is interested in studying whether employers' costs of providing these benefits are shifted to the workers likely to need them. Specifically, the paper studies whether employers are able to pass on group-specific costs on the basis of demographics, or whether other frictions, such as anti-discrimination legislation, prevent this and make some workers more costly for employers to hire.
The paper exploits two main sources of policy variation: state-level and federal-level mandates which required insurance companies to cover childbirth costs on a basis equal to their coverage of other medical conditions. In 1978, the federal Pregnancy Discrimination Act (PDA) made this requirement national.
The author uses a triple difference design and is interested in comparing multiple subgroups. The subgroups of interest are those more likely to use the coverage for childbirth costs: married women age 20-40, single women age 20-40, and married men age 20-40 (whose may have wives covered by their insurance policy). The comparison individuals are single men age 20-40 and people over age 40, who are unlikely to use these benefits. The parallel gaps assumption requires that, in the absence of the mandates, the difference between the comparison subgroup and each of the subgroups of interest would have evolved similarly across states. That is, for example, any macroeconomic or state-level shocks correlated with the policy would have affected the subgroups similarly.
The data for this analysis are drawn from the May Current Population Survey 1974-1978 usbureauofthecensusCurrentPopulationSurvey1992a,usbureauofthecensusCurrentPopulationSurvey1992b,usbureauofthecensusCurrentPopulationSurvey1992c,usbureauofthecensusCurrentPopulationSurvey1992. In my re-analysis, I focus on the state-level mandates. Between July 1, 1976 and January 1, 1976, three states enacted such a mandate to provide coverage for childbirth. Thus, for this analysis, the treated states are Illinois, New Jersey, and New York. The untreated comparison states are Ohio, Indiana, Connecticut, Massachusetts, and North Carolina. The study window covers two years before the mandate (1974 and 1975) and two years after (1977 and 1978).\footnote{For simplicity in this analysis, and consistent with the original paper, I pool these into a pre- and post-period.}
The author's conclusions suggest that employers can, and do, pass on group-specific costs to workers. The paper first documents that, before these mandates came into effect, many people did not have full coverage for the costs of childbirth and that adding this policy to insurance packages was likely to be very costly. The author then attempts to evaluate causal impacts of the mandates on the hourly wages, hours worked, and employment of the subgroups. The author concludes that, for targeted workers likely to use the childbirth coverage, wages decreased significantly.
In this section, I evaluate the finite-sample performance of the proposed estimators for the CDATT using a Monte Carlo simulation study. I design a setup calibrated to the empirical application in gruberIncidenceMandatedMaternity1994 to highlight a clear case where the DATT and CDATT diverge and in which failing to differentiate the two when interpreting results could lead to misleading conclusions. I show the performance of my estimator under different forms of treatment effect heterogeneity.
Let $Y$ represent the log of hourly wage, $W$ represent whether the individual lives in a state with a mandate or no mandate, $S$ represent whether the individual is a member of the targeted subgroup (married and single women age 20-40, as well as married men age 20-40) or a member of the untargeted subgroup (for this simulation, unmarried men age 20-40),\footnote{Although the original design also includes individuals over age 40 in the untargeted group, these are excluded in this simulation in order to improve the overlap between the groups with respect to age.} and $X$ represent the individual's covariates (education, a quadratic in age, white/non-white, union/non-union, and white-collar/not white collar job).
The simulation sample is constructed as follows: for individual $i$, a vector $X_i$ is drawn randomly from the data, without replacement, so that the original distribution of covariates is maintained. To match the formal results outlined above on panel data, a panel is created by taking all pre-period observations and assuming that covariates do not change over time.
Then, subgroup status is assigned according to the following propensity score models:
The coefficients $\alpha^{ws}, \alpha^{ws'},$ and $\alpha^{cs}$ are chosen to create a realistic data generating process (DGP) by using the coefficients for the same regression in the original, unaltered dataset. The intercept is omitted to maintain balance in the share belonging to each subgroup.
Then, outcomes are constructed according to an outcome regression:
with $Y_0$ assigned to observations in the pre-period and $Y_1$ assigned to observations in the post-period. Again, the coefficients $\beta_0$ and $\beta$ are chosen to give a realistic DGP by using the coefficients for the same regression in the unaltered, original dataset. The error terms $u_i^{pre}, u_i^{post} \sim \mathcal{N}(0, \sigma_u)$, where $\sigma_u$ is the standard deviation of the residuals of this regression in the original dataset.
The crucial feature of this outcome regression is $R_i$, which represents a random variable distributed according to $\mathcal{N}(\beta X_i, \gamma \sigma_{\beta X_i})$, where $\sigma_{\beta X_i}$ is the standard deviation of $\beta X_i$ across observations. In this setup, the $W_i R_i$ term introduces a non-zero average treatment effect on the treated (because outcomes depend causally on $W_i$), the magnitude of which is heterogeneous across individuals (because $R_i$ is random) and, importantly, across subgroups (because $X_i$ is correlated with $R_i$ and $S_i$ given the propensity score models). The parameter $\gamma$ controls the strength of these correlations. For comparison, I also show a scenario where this term is omitted.
This setup clearly showcases a case where the DATT and CDATT differ. The outcome model is constructed so that the true CDATT is zero, because there is no causal effect of subgroup status. However, the true DATT is not zero, because there is a causal effect of treatment $W$ which differs between the subgroups. More formally, for each individual,
Thus, individual treatment effects are given by
which is correlated with $S_i$ via $X_i$.
Given this setup, the outcome regressions are correctly specified when they are linear in $X_i$ and the propensity score models are correctly specified by a multinomial logistic regression. For comparison, examples will be shown when these models are misspecified. When models are misspecified, the only covariate used to estimate the parameters of interest will be a nonlinear function of education, where
That is, the models specified by the researcher will be misspecified because they omit the other elements of $X_i$ and because of the nonlinear transformation of education.
In this section, I highlight the properties of my proposed estimators in the calibrated simulation described above. The simulations highlight a clear case where the DATT and CDATT can diverge. They show how the performance of the estimators for the DATT and CDATT compare depending on the variance in treatment effects. Finally, I also highlight the properties of the doubly-robust estimator for the CDATT and show that, even in finite samples, it achieves its semiparametric efficiency bound.
Figure (ref) introduces the scenarios studied, each with a different distribution of treatment effects, controlled by the parameter $\gamma$. As described above, $\gamma$ controls the variance of the distribution of treatment effects between individuals. The figures highlight that, as $\gamma$ increases and the variance of treatment effects increases, the overlap between subgroups becomes larger and the difference between subgroups becomes more difficult to distinguish.
Table (ref) and Figure (ref) highlight two key features of the performance of the estimators for the DATT and the CDATT under the various scenarios for treatment effect heterogeneity.
First, the results underscore the divergence between the DATT and the CDATT when the underlying average treatment effects on the treated are correlated with subgroup status. When there is no difference in the treatment effects between the groups, the estimates of the DATT and CDATT are virtually identical. However, when the treatment effects vary with subgroup status, the DATT differs strongly from the CDATT. For example, an analysis of the DATT could lead a researcher to conclude that the treatment caused individuals in the targeted subgroup to experience an approximately 0.2 log point increase in wages relative to their untargeted counterparts. However, an analysis of the CDATT would highlight that this differential impact on the targeted subgroup was not causally due to their targeted-subgroup status, but rather explained by other factors, like age and education, that differ between the subgroups. The simulations make clear that, regardless of which estimator is used for the DATT, conditioning on covariates does not help the researcher recover an estimate of the CDATT.
Second, the results show that, as the variance in treatment effects increases, the estimators lose power and precision. When $\gamma = 5$, the variance of the underlying treatment effects is large, and both estimators have substantially larger standard errors and confidence intervals relative to the other scenarios. The loss in power is somewhat greater for the estimators of the CDATT due to the increased difficulty of distinguishing between the two subgroups. However, under moderate levels of treatment effect variance, the estimators of the CDATT have standard errors and confidence intervals on par with the DATT.
Next, Table (ref) and Figure (ref) compare the performance of the different estimators of the CDATT. The results highlight both the double-robustness of the doubly-robust estimator and its semiparametric efficiency.
The results show that the doubly-robust estimator of the CDATT performs well when at least one (but not necessarily both) set of working models is correctly specified. When both sets of working models are correctly specified, as in Case 1 in Table (ref), the regression adjustment, IPW, and doubly-robust estimators all perform well, with minimal bias. The doubly-robust estimator attains nearly correct coverage of the 95% confidence interval. When only the outcome regressions are specified correctly but the propensity scores are misspecified, as in Case 2, the regression adjustment and doubly-robust estimators have minimal bias, while the IPW estimator is biased. On the other hand, when the propensity score models are correct but the outcome regressions are misspecified, as in Case 3 and as expected, the IPW and doubly-robust methods are nearly unbiased, while the regression adjustment estimator has substantial bias. When all working models are misspecified, all available estimators have substantial bias and are not efficient.
The results also demonstrate the desirable performance of the doubly-robust estimator semiparametric efficiency of the parameter. Across all cases, the standard error implied by the efficiency bound is 0.085. When both the propensity score models and the outcome regressions are correctly specified, the doubly-robust estimator achieves this efficiency bound. The regression adjustment estimator slightly outperforms the efficiency bound, while the IPW estimator is less efficient, a result paralleled by and discussed in santannaDoublyRobustDifferenceindifferences2020. When the outcome regressions are correctly specified, the doubly-robust estimator remains nearly as efficient as when the working models are correctly specified, this is not the case when the propensity score models are misspecified. For more details on doubly-robust estimators that are also asymptotically efficient under misspecification of either the propensity score or outcome model, see santannaDoublyRobustDifferenceindifferences2020.
As in the original paper, I use a triple difference design to estimate the impacts of coverage for maternity benefits on three subgroups of interest relative to a subgroup unlikely to use these benefits. I extend the analysis by implementing my proposed estimators of the CDATT.
The specification in the paper is as follows:
where $Y_{ijt}$ represents the outcome for individual $i$ in state $j$ in year $t$, $\delta_j$ represents state fixed effects, $\tau_t$ represents year fixed effects, $s_i$ represents the demographic subgroup, and $\times$ represents the interaction between two variables. Control variables are represented by $X_{ijt}$ and include education, experience and experience$^2$, sex, marital status, an interaction between sex and marital status, a binary variable for white/non-white, union/non-union, and indicators for 15 major industries. The coefficient of interest is given by $\beta_8$.
To motivate the inclusion of covariates and their role in this analysis, I first highlight the divergence between demographic groups in these covariates. Figure (ref) shows, for example, that single women age 20-40 tend to have higher education than the other demographic groups, are more likely to be non-white, and are more likely to work in a white-collar job. This motivates the desire to separate between the DATT and CDATT parameters. If certain jobs, such as white-collar jobs, are more or less sensitive to the mandated benefits, then any difference in ATTs between the demographic groups might be explained by their different job characteristics rather than their gender, age, or marital status.
As described above, the DATT and the CDATT coincide under the assumption that any untargeted workers would be unaffected by the policy. That is, if single men age 20-40 and people over age 40 are assumed to be entirely unaffected by the coverage of childbirth costs, regardless of other characteristics, then these two parameters will be equivalent. However, this assumption does not seem justified. For example, if the mandates make some workers more desirable (less costly) than others, then there will be spillovers on this untargeted group as employers substitute towards them. Further, if employers cannot pass on group-specific costs, then outcomes for all workers will be affected by the mandates.
In my analysis, I use the following covariates: education, bins for age groups (under 25, 26-30, 31-35, and over 35), white/non-white, and white-collar/not white-collar.
Estimating the DATT, as in the published paper, I find that the impacts of mandating childbirth coverage differ between the targeted subgroups (married women age 20-40, married men age 20-40, and single women age 20-40). However, when estimating the CDATT, I find evidence that these differences between the subgroups should not be interpreted as caused by subgroup status, but instead appear to be due to differences in covariates between the groups.
Figure (ref) highlights that, when estimating the DATT on the log of hourly wages, results are consistent with the original published results, regardless of which estimator of the DATT is used. When not including controls, the CDATT and the DATT coincide. When using a 3WFE design similar to that in the original paper, the results suggest an impact of -0.009, -0.039, and -0.079 log points in wages for married men age 20-40, married women age 20-40, and single women age 20-40, respectively. These estimates are similar to those presented in the original paper, which suggested impacts of -0.009, -0.043, and -0.042 for these groups, respectively (see Table 4 in gruberIncidenceMandatedMaternity1994). When including controls using the doubly robust estimator, the magnitudes of the estimates of the DATT are largely unchanged. These estimates suggest a reduction in wages for all targeted groups, which is significant regardless of estimator for single women age 20-40. The published result found significant declines for both married women age 20-40 and single women age 20-40.
Although analysis of the DATT suggests that the benefits mandates reduced wages for the targeted subgroups relative to the untargeted subgroup, the CDATT suggests this may not be causally due to subgroup status. Figure (ref) shows how the estimates of the CDATT diverge from the DATT. For married men, the doubly-robust estimate is large and positive (although insignificant). For married women, it is slightly positive (insignificant), and for single women, it is slightly negative (insignificant). Although the DATT is consistently negative for all three groups, the CDATT offers a qualitatively different result, with impacts that are less negative and are insignificant.
Turning to the impacts on hours worked, the DATT and the CDATT again may have different interpretations. For married men age 20-40, the 3WFE specification reveals a significant increase in hours worked per week, by about 0.044 log points (compared to the published estimate of 0.030 log points). The magnitude of this impact is similar when including controls via the doubly-robust estimator of the DATT (0.047 log points). However, it is nearly zero when estimating the CDATT using the doubly-robust estimator. For married women, again, both estimates of the DATT reveal a positive impact of 0.058 and 0.059 log points (compared to the published impact of 0.049 log points). However, the estimate of the CDATT is smaller, only 0.019 log points, and insignificant. Finally, there is no significant impact in any case for single women, as in the published results.
The results on employment suggest less divergence between the two estimators, and weaker overall evidence of an impact. As in the published paper, estimates of the DATT are generally negative and insignificant. However, it is not surprising that the DATT and CDATT diverge less in this case. The difference between the two arises when treatment effects are correlated with subgroup status, and, if treatment effects are truly small or null, the magnitude of the difference between subgroups will be smaller.
In this paper, I have addressed identification and estimation in triple difference designs when treatment effects are heterogeneous. I begin by discussing two parameters of interest, the difference in ATTs between subgroups (DATT) and the causal difference in ATTs between subgroups (CDATT). When treatment effects are heterogeneous, these two parameters may differ in important ways, and caution is warranted to avoid interpreting a DATT as a CDATT.
I show that the DATT can be identified under an assumption on the trend in the gap between subgroups' outcomes. I highlight that the ATT can be recovered only under the assumption of a subgroup that is unaffected by the treatment. I then show that identification of the CDATT requires additional assumptions on treatment effect heterogeneity, for example, the assumption that treatment effect heterogeneity is captured by observable characteristics.
Next, I derive the semiparametric efficiency bounds for the CDATT, and propose estimators for it. I discuss their asymptotic properties and show that these estimators achieve semiparametric efficiency. A realistic simulation study calibrated to the application in gruberIncidenceMandatedMaternity1994 highlights these results. The simulations first highlight the divergence between the DATT and the CDATT, and show that estimators for the DATT can provide misleading results if they are interpreted causally in cases where subgroups differ in their sensitivity to the treatment. They also show that the proposed estimators for the CDATT are unbiased and perform well in finite samples.
An application of these estimators to the analysis in gruberIncidenceMandatedMaternity1994 highlights a case where heterogeneity in sensitivity to a treatment can correlate with subgroup status, affecting the interpretation of the triple difference estimates. The analysis addresses the question of whether employers can pass on group-specific costs on the basis of demographic characteristics. The paper uses a triple difference design to study the impacts of state-level mandates requiring insurance to cover costs for childbirth. Since these benefits are likely to be used by certain groups (ie, women age 20-40 and married men age 20-40), the triple difference design allows to test for differential effects for this group relative to other segments of the population. Although an analysis of the DATT suggests that there is significant differential cost-shifting for these groups, estimating the CDATT offers little evidence that this difference is causal on the basis of demographic characteristics. Researchers should take treatment effect heterogeneity into account in the interpretation and estimation of treatment effects in triple difference designs.
\printbibliography