The exact contents of citations.db main_text.text for this paper — one flattened LaTeX string, title through conclusion, appendix excluded, unmodified except for removing email addresses. This is what our citation measures are computed over.
76,292 characters
The Experimental Selection Correction Estimator: Using Experiments to Remove Biases in Observational Estimates
\title{
\Large The Experimental Selection Correction Estimator:\\
\Large Using Experiments to Remove Biases in Observational Estimates\thanks{{\small An earlier version was circulated under the title ``Combining Experimental and Observational Data to Estimate Treatment Effects on Long Term Outcomes." We thank Kevin Chen and Yechan Park for excellent research assistance. We are grateful for comments from Gary Chamberlain, Nathan Kallus, Xiaojie Mao, David Ritzwoller, Jonah Rockoff, Dylan Small, and numerous seminar participants. This research was funded through the Chan-Zuckerberg Initiative, Sloan Foundation, Schmidt Futures, ONR grant N00014-17-1-2131 and N00014-19-1-2468, Harvard University, and a gift from Amazon.}} }
\author{Susan Athey\thanks{{\small Graduate School of Business, Stanford University and NBER,
[email removed]. }} \and Raj Chetty\thanks{{\small Department of
Economics, Harvard University, Opportunity Insights, and NBER,
[email removed]. }}
\and Guido W. Imbens\thanks{{\small
Graduate School of Business, Stanford University and NBER,
[email removed]. }}
}
\date{First Version: August 2019; Current version: \ifcase\month\or
January\or February\or March\or April\or May\or June\or
July\or August\or September\or October\or November\or December\fi \ \number
\year\ \ }
\maketitle
\thispagestyle{empty}
\begin{abstract}
\noindent Researchers increasingly have access to two types of data: (i) large observational datasets where treatment ({\it e.g.}, class size) is not randomized but several primary outcomes ({\it e.g.}, graduation rates) and secondary outcomes ({\it e.g.}, test scores) are observed and (ii) experimental data in which treatment is randomized but only secondary outcomes are observed. We develop a new method to estimate treatment effects on primary outcomes in such settings. We use the difference between the secondary outcome and its predicted value based on the experimental treatment effect to measure selection bias in the observational data. Controlling for this estimate of selection bias yields an unbiased estimate of the treatment effect on the primary outcome under a new assumption that we term \textit{latent unconfoundedness}, which requires that the same confounders affect the primary and secondary outcomes. Latent unconfoundedness weakens the assumptions underlying commonly used surrogate estimators. We apply our estimator to identify the effect of third grade class size on students’ outcomes. Estimated impacts on test scores using OLS regressions in observational school district data have the opposite sign of estimates from the Tennessee STAR experiment. In contrast, selection-corrected estimates in the observational data replicate the experimental estimates. Our estimator reveals that reducing class sizes by 25\% increases high school graduation rates by 0.7 percentage points. Controlling for observables does not change the OLS estimates, demonstrating that experimental selection correction can remove biases that cannot be addressed with standard controls.
\end{abstract}
\vskip0.5cm
Keywords:\ causality, surrogates, observational studies, long-term outcomes, control functions
\baselineskip=20pt\newpage \setcounter{page}{1}
\newpage
\section{Introduction}
\label{section:introduction}
As observational data become more widely available, researchers seeking to estimate treatment effects increasingly have access to two types of data: (i) large observational datasets where treatments and a broad range of outcomes are observed, but treatment is not randomized and (ii) smaller experimental datasets where treatment is randomly assigned, but only a subset of outcomes are observed. For example, in the context of education, many analysts have been interested in identifying the causal effects of classroom sizes in elementary school on high school graduation rates. Observational data with information on class sizes and graduation rates are now widely available from school districts' administrative records. But causal inference using these data is challenging because of selection biases arising from non-random assignment to classrooms. Causal inference is more straightforward in experimental data – such as the widely studied Project STAR class size experiment (e.g., \citet{krueger1999experimental}) – but experimental datasets often do not contain information on outcomes such as graduation rates because they are observed with long lags.
The most common method of identifying the causal effect of a treatment ({\it e.g.}, class size reduction) on the primary outcome of interest ({\it e.g.}, graduation rates) in such settings is to use secondary intermediate outcomes that are observed in the experimental data ({\it e.g.}, test scores) as statistical surrogates \citep{prentice1989surrogate, athey2019surrogate}. The surrogate approach, illustrated in Figures 1a-b below, uses the observational dataset to estimate the relationship between the primary outcome ($Y^{\rm P}_i$) and secondary outcome ($Y^{\rm S}_i$), and then estimates the impact of the treatment of interest ($W_i$) on $Y^{\rm P}_i$ based on that relationship. Under the surrogacy assumptions that (i) $W_i$ only affects $Y^{\rm P}_i$ through its impact on $Y^{\rm S}_i$ and (ii) there are no unobserved confounders that affect the relationship between $Y^{\rm S}_i$ and $Y^{\rm P}_i$ in the observational sample, this approach provides an unbiased estimate of the effect of $W_i$ on $Y^{\rm P}_i$.
The surrogate approach has been applied in many fields, from economics to product testing to public health \citep{alonso2006unifying, adams2006overweight, d2006surrogate, gupta2019top}. Yet there remains concern that the surrogacy assumptions may be violated in these settings. For example, test scores are a widely used surrogate in labor economics, but researchers have identified other pathways through which childhood interventions affect long-term outcomes outside test scores, such as non-cognitive skills \citep{heckman2006effects, chetty2011does}.
\begin{figure}[H]
\centering
\caption{Comparison of Experimental Selection Correction and Surrogate Models}
\captionsetup[subfigure]{labelformat=simple, labelsep=period, position=top, justification=centering}
\begin{subfigure}[b]{0.8\textwidth}
\centering
\caption{\scalebox{0.9}{\textbf{Experimental Data in Surrogate Model}}}
\resizebox{\linewidth}{!}{
\begin{tikzpicture}[
>=Stealth,
node distance=1.8cm and 5.4cm
]
\node[draw, circle, fill=customorange, inner sep=0mm, outer sep=0mm,
minimum size=5mm, label=above:{{\scalebox{1.3}{\small Test Scores ($Y_i^{\rm S}$)}}}] (2) {};
\node[observed, minimum size=5mm, label=left:{{\scalebox{1.3}{\small Class Size ($W_i$)}}}] (1) [left=of 2] {};
\node[unobserved_h, minimum size=5mm, label=right:{{\scalebox{1.3}{\small HS Graduation ($Y_i^{\rm P}$)}}}] (3) [right=of 2] {};
\draw [-{Stealth[scale=1.2]}, shorten <=5pt, shorten >=5pt] (1.east) -- (2.west);
\draw [-{Stealth[scale=1.2]}, shorten <=5pt, shorten >=5pt] (2.east) -- (3.west);
\draw[line width=0.8pt] (3.south west) -- (3.north east);
\end{tikzpicture}
}
\end{subfigure}
\vspace{30pt}
\begin{subfigure}[b]{0.8\textwidth}
\centering
\caption{\scalebox{0.9}{\textbf{Observational Data in Surrogate Model}}}
\label{fig:1b}
\resizebox{\linewidth}{!}{
\begin{tikzpicture}[
>=Stealth,
node distance=1.8cm and 5.4cm
]
\node[draw, circle, fill=customorange, inner sep=0mm, outer sep=0mm,
minimum size=5mm, label=above:{{\scalebox{1.3}{\small Test Scores ($Y_i^{\rm S}$)}}}] (2) {};
\node[unobserved_c, minimum size=5mm, label=left:{{\scalebox{1.3}{\small Class Size ($W_i$)}}}] (1) [left=of 2] {};
\node[circle, fill=customgreen, inner sep=0mm, outer sep=0mm,
minimum size=5mm, draw=black, label=right:{{\scalebox{1.3}{\small HS Graduation ($Y_i^{\rm P}$)}}}] (4) [right=of 2] {};
\draw [-{Stealth[scale=1.5]}, shorten <=8pt, shorten >=8pt] (1.east) -- (2.west);
\draw [-{Stealth[scale=1.5]}, shorten <=8pt, shorten >=8pt] (2.east) -- (4.west);
\draw[line width=0.8pt] (1.south west) -- (1.north east);
\end{tikzpicture}
}
\end{subfigure}
\vspace{10pt}
\vspace{30pt}
\begin{subfigure}[b]{0.8\textwidth}
\centering
\caption{\scalebox{0.9}{\textbf{Experimental Data in Experimental Selection Correction (ESC) Model}}}
\resizebox{\linewidth}{!}{
\begin{tikzpicture}[
>=Stealth,
node distance=1.8cm and 5.4cm
]
\node[draw, circle, fill=customorange, inner sep=0mm, outer sep=0mm,
minimum size=5mm, label=above:{{\scalebox{1.3}{\small Test Scores ($Y_i^{\rm S}$)}}}] (2) {};
\node[observed, minimum size=5mm, label=left:{{\scalebox{1.3}{\small Class Size ($W_i$)}}}] (1) [left=of 2] {};
\node[unobserved_h, minimum size=5mm, label=right:{{\scalebox{1.3}{\small HS Graduation ($Y_i^{\rm P}$)}}}] (3) [right=of 2] {};
\draw [-{Stealth[scale=1.2]}, shorten <=5pt, shorten >=5pt] (1.east) -- (2.west);
\draw [-{Stealth[scale=1.2]}, shorten <=5pt, shorten >=5pt] (2.east) -- (3.west);
\draw [-{Stealth[scale=1.2]}, shorten <=5pt, shorten >=5pt] (1.north east) to [bend left=20] (3.north west);
\draw[line width=0.8pt] (3.south west) -- (3.north east);
\end{tikzpicture}
}
\end{subfigure}
\vspace{30pt}
\begin{subfigure}[b]{0.8\textwidth}
\centering
\caption{\scalebox{0.9}{\textbf{Observational Data in Experimental Selection Correction (ESC) Model}}}
\resizebox{\linewidth}{!}{
\begin{tikzpicture}[
>=Stealth,
node distance=1.8cm and 5.4cm
]
\node[draw, circle, fill=customorange, inner sep=0mm, outer sep=0mm,
minimum size=5mm, label=above:{{\scalebox{1.3}{\small Test Scores ($Y_i^{\rm S}$)}}}] (2) {};
\node[observed, minimum size=5mm, label=left:{{\scalebox{1.3}{\small Class Size ($W_i$)}}}] (1) [left=of 2] {};
\node[circle, fill=customgreen, inner sep=0mm, outer sep=0mm,
minimum size=5mm, draw=black, label=right:{{\scalebox{1.3}{\small HS Graduation ($Y_i^{\rm P}$)}}}] (4) [right=of 2] {};
\node[unobserved, minimum size=5mm, label=below:{{\scalebox{1.3}{\small Unobserved Confounder ($U_i$)}}}] (3) [below=of 2] {};
\draw [-{Stealth[scale=1.5]}, shorten <=10pt, shorten >=10pt] (1.east) -- (2.west);
\draw [-{Stealth[scale=1.5]}, shorten <=10pt, shorten >=10pt] (2.east) -- (4.west);
\draw [-{Stealth[scale=1.5]}, shorten <=10pt, shorten >=10pt]
(1.north west) to[out=30,in=150] (4.south west);
\draw [dashed, -{Stealth[scale=1.5]}, shorten <=10pt, shorten >=10pt] (3.north) -- (2.south);
\draw [dashed, -{Stealth[scale=1.5]}, shorten <=10pt, shorten >=10pt] (3.north west) -- (1.south east);
\draw [dashed, -{Stealth[scale=1.5]}, shorten <=10pt, shorten >=10pt] (3.north east) -- (4.south west);
\draw[line width=0.8pt] (3.south west) -- (3.north east);
\end{tikzpicture}
}
\end{subfigure}
\vspace{0.2cm}
\caption*{\footnotesize
\textit{Notes:}
This figure depicts the assumptions and informational structures underlying the surrogate and Experimental Selection Correction (ESC) estimators using directed acyclic graphs. Solid circles denote variables observed in the data; empty circles with lines denote unobserved variables. Panels A and B depict the experimental and observational data in the surrogate approach, while Panels C and D show the same for the ESC estimator. In the experimental data (Panels A and C), class size \(\bigl(W_i\bigr)\) is randomly assigned, ensuring that there is no unobserved confounder when estimating the treatment effect on test scores \(\bigl(Y_i^S\bigr)\). However, the primary outcome of interest, high school graduation \(\bigl(Y_i^P\bigr)\), is unobserved in the experimental sample. The ESC estimator requires that \(\,W_i\) is observed in the observational sample, whereas the surrogate estimator does not. The identifying assumptions underlying the surrogate estimator are that (i) any effect of \(\,W_i\) on the primary outcome \(\,Y_i^P\) operates exclusively through the secondary outcome \(\,Y_i^S\) (Panels A and B) and (ii) there are no unobserved confounders that affect the relationship between $Y^{\rm S}_i$ and $Y^{\rm P}_i$ in the observational sample (Panel C). The ESC estimator relaxes these assumptions by (i) permitting a direct effect of \(\,W_i\) on \(\,Y_i^P\) (Panels C and D) and (ii) allowing for an unobserved confounder \(\bigl(U_i\bigr)\) that influences both class size and graduation outcomes in the observational data (Panel D).
}
\end{figure}
In this paper, we develop an “Experimental Selection Correction” (ESC) estimator that identifies the effect of $W_i$ on $Y^{\rm P}_i$ even when the surrogacy assumptions are violated, as illustrated in Panels C and D of Figure 1. Our estimator relies on more information than the surrogacy approach: it requires that the observational dataset contains information not just on the primary and secondary outcomes, $Y^{\rm S}_i$ and $Y^{\rm P}_i$, but also on treatment $W_i$ (with variation in treatment across observations). With this additional information, we show how one can identify the effect of $W_i$ on $Y^{\rm P}_i$ under strictly weaker assumptions than those required for the surrogacy approach.
To illustrate the general information scheme we analyze, consider a setting with two datasets: (i) the Project STAR experimental data, where class size is randomized and we observe test scores ($Y^{\rm S}_i$) but not high school graduation rates ($Y^{\rm P}_i$), and (ii) observational data from the New York City school district, in which class size is observed but not randomized (and hence likely to be correlated with both observed and unobserved characteristics) and we observe both test scores and graduation rates.
In the STAR experimental data, we can estimate the treatment effect of small class size ($W_i$) on 3rd grade test scores by regressing test scores on an indicator for being assigned to a small class (with 7 fewer students on average) in 3rd grade. Column 1 of Table 1 shows that being assigned to a small class in 3rd grade increases students’ end-of-3rd-grade test scores by 0.19 standard deviations (SD). We cannot, however, estimate the effect of class size on high school graduation in the STAR data, because we do not observe graduation in the STAR sample.\footnote{Researchers attempted to follow the STAR students longitudinally, but were only able to collect information on high school graduation for 43\% of students, whose characteristics are not representative of the experimental sample as a whole. This underscores the challenges of tracking primary outcomes in experiments and motivates the approach we take here of combining observational administrative records and experimental data.}
\begin{table}[tbh]
\centering
\caption{Estimated Effects of Small Class Assignment in STAR vs. NYC Data}
\label{tab:1}
\vspace{1em}
\resizebox{0.88\textwidth}{!}{
\setlength{\tabcolsep}{6pt}
\begin{tabular}{lccc}
\toprule
Sample: & Exp.\,(STAR) & Obs.\,(New York) & Exp.\,+\,Obs.\,(STAR + NYC) \\
Estimator: & OLS & OLS & Exp.\ Selection Correction (ESC) \\[0.2cm]
Outcome & & & \\
\midrule
3rd Grade Test Score & \begin{tabular}[c]{@{}c@{}}$0.19$\\$(0.04)$\end{tabular}
& \begin{tabular}[c]{@{}c@{}}$-0.12$\\$(0.01)$\end{tabular}
& \begin{tabular}[c]{@{}c@{}}$0.19$\\$(0.04)$\end{tabular} \\[0.5em]
(secondary outcome) \\[0.5em]
HS Graduation & ?
& \begin{tabular}[c]{@{}c@{}}$-1.76$\\$(0.29)$\end{tabular}
& \begin{tabular}[c]{@{}c@{}}$0.69$\\$(0.34)$\end{tabular} \\
(primary outcome) \\
\bottomrule
\end{tabular}
}
\vspace{0.2cm}
\resizebox{0.88\textwidth}{!}{
\begin{tabular}{p{0.95\textwidth}}
\footnotesize{
\textit{Notes:} This table reports point estimates (with standard errors in parentheses) of the effect of assignment to a small class on end-of-3rd-grade test scores and high school graduation rates. Each cell reports estimates from a separate model. Columns 1 and 2 present ordinary least squares (OLS) estimates using the experimental STAR sample and the observational New York sample, respectively. Column 3 combines both experimental and observational data using the Experimental Selection Correction (ESC) estimator. The missing entry denoted by “?” for High School Graduation in the STAR sample reflects the absence of information on graduation in the STAR dataset. The specification in Column 1 includes school fixed effects (since all STAR students are in the same cohort), while Columns 2 and 3 include both school and cohort fixed effects. }
\end{tabular}
}
\end{table}
In the observational NYC sample, estimating an analogous OLS regression of test scores on an indicator for being assigned to a small class yields an estimate of $-0.12$ SD (s.e. 0.01, Column 2 of Table 1). Children in smaller classes are also 1.76 percentage points (s.e. 0.29) less likely to graduate from high school. These negative estimates of the causal effect of class size reductions are implausible both in the light of the positive experimental Project STAR estimates and based on {\it a priori} beliefs. Of course, the OLS estimates may be confounded because class size is not randomly assigned in NYC. For example, students with needs for additional educational support may be assigned to smaller classes. Our goal is to obtain an unconfounded estimate of $W_i$ on $Y^{\rm P}_i$, {\it i.e.}, to fill in the lower left box in Table 1.
We obtain an unconfounded estimate of the effect of $W_i$ on $Y^{\rm P}_i$ in the observational data by using the difference in the distribution of test scores ($Y^{\rm S}_i$) conditional on treatment in the observational and experimental samples to adjust for selection. In linear models, the ESC estimator can be implemented in three straightforward steps (see Appendix for code). First, we estimate the effect of class size on test scores ($\tau^{\rm S}$) in the experimental sample using a linear regression, as in Column 1 of Table 1.\footnote{We focus on the case where treatment is randomly assigned in an experiment, but any quasi-experimental research design that yields an unbiased estimate of the treatment effect on the secondary outcome $\tau^{\rm S}$ can be used to implement the ESC estimator.} Second, for all students in the observational sample, we calculate the difference between the secondary outcome (test score) and the predicted test score based on the student’s class size (the residual $\alpha^{\rm S}_i= Y^{\rm S}_i - \tau^{\rm S} W_i$), with the parameter $\tau^{\rm S}$ in the prediction model coming from the experimental sample. Finally, we regress the primary outcome (graduation rates) on treatment (class size) in the observational data, controlling for the residual $\alpha^{\rm S}_i$.
We show that this control function approach identifies the causal effect of $W_i$ on $Y^{\rm P}_i$ under three assumptions: (i) random assignment (or unconfoundedness) in the experimental sample; (ii) a standard external validity assumption; and (iii) a new assumption that we term \textit{latent unconfoundedness}. External validity requires that (conditional on pretreatment observables), the treatment effect in the experimental sample is the same as the treatment effect in the population represented by the observational sample \citep{shadishcookcampbell, hotz2005predicting}. Latent unconfoundedness requires that the unobserved confounders that affect the primary outcome (graduation rates) are the same as those that affect the secondary outcome (test scores). Under this assumption, the difference between the actual secondary outcome in the observational data and the predicted secondary outcome based on the experimental estimate ($\alpha^{\rm S}_i$) fully captures any selection bias that affects the primary outcome. Thus, controlling for $\alpha^{\rm S}_i$ is sufficient to identify the causal effect of $W_i$ on $Y^{\rm P}_i$. Intuitively, $\alpha^{\rm S}_i$ functions as a selection correction, similar to parametric selection correction approaches dating to \citet{heckman1979sample} and control function methods (\citet{heckman1985alternative, imbens2009identification, wooldridge2015control}).
The main theoretical result of this paper is that the treatment effect of $W_i$ on $Y^{\rm P}_i$ is point-identified under latent unconfoundedness, external validity, and random assignment in the experimental sample (without any functional form or distributional assumptions). We also present a control function approach to estimation for the general, nonlinear case. A corollary of our main result is that if an observational estimate of the treatment effect on the secondary outcome ($Y^{\rm S}_i$) is the same as the experimental estimate, then under linearity, latent unconfoundedness and external validity together imply that the observational estimator is unconfounded for the primary outcome. Many empirical studies show that an observational and experimental estimator yield similar estimates for secondary outcomes and then use this as a heuristic justification for estimating impacts on a broader range of primary outcomes using the observational estimator ({\it e.g.}, \citet{chetty2014onemeasuring, bleemer2022affirmative}). Our analysis makes precise the conditions—most importantly, latent unconfoundedness across outcomes—under which this heuristic is justified.
We also propose falsification tests for the underlying identifying assumptions that make use of additional “holdout” post-treatment outcomes ($Y^H_i$) observed in both datasets. One could use these measures as additional secondary outcomes. An alternative is to use them to validate the estimator instead of using them to implement the selection correction itself. Tests of whether our ESC estimator that uses 3rd grade scores for selection correction matches experimental estimates for 4th and 5th grade test scores (holdout outcomes) serve as tests for whether the underlying latent unconfoundedness and external validity assumptions jointly hold.
We apply the experimental selection correction estimator to estimate the causal effects of 3rd grade class size reduction on high school graduation rates in the New York City data, using end-of-3rd-grade test scores as the secondary outcome for the selection correction. We use the Tennessee STAR sample as the experimental sample in which we estimate the treatment effect of class size reduction on 3rd grade test scores.\footnote{Naturally, one may have concerns about the external validity of the Tennessee sample for the New York City data. Both samples reflect a relatively low-income population with fairly similar demographic characteristics. Furthermore, we show that adjusting for remaining differences in observable demographics does not affect our conclusions meaningfully.}
\begin{figure}[H]
\centering
\caption{Estimated Treatment Effects of Assignment to Small Class in 3rd Grade}
\includegraphics[width=1\linewidth]{fig_main_int3.pdf}
\label{fig_lu_main}
\caption*{\footnotesize
\textit{Notes:} This figure plots estimates of the effect of being assigned to a small class in 3rd grade on two outcomes: standardized test scores (left panel) and high school (HS) graduation rates (right panel). In the left panel, effect sizes are reported in standard deviations of test scores; in the right panel, they are reported in percentage points of HS graduation. The “Experimental Estimate” series presents estimates from OLS regressions with school fixed effects in the STAR data. The “Observational OLS” estimate is obtained from OLS regressions with school and cohort fixed effects in the NYC observational data. The “Experimental Selection Correction” (ESC) estimate adjusts the observational OLS estimate using a selection correction term estimated using the experimental sample as described in the text. Vertical bars denote 95\% confidence intervals.}
\end{figure}
As discussed above, standard OLS regression estimates of 3rd grade test scores on an indicator for small class size in the NYC data yield a treatment effect estimate of $-0.12$ SD. The ESC estimator (shown in Column 3 of Table 1) yields an estimate of $0.19$ SD, coinciding with the STAR experimental estimate by construction. The ESC treatment effect estimates for test scores in grades 4-8 (holdout outcomes) are nearly identical to the STAR experimental estimates (Figure \ref{fig_lu_main}). Notably, they capture the well-known “fadeout” pattern on test score impacts documented in prior work (\cite{deming2009early,chetty2011does,cascio2012knowledge}). These results support the identification assumptions underlying our method and more broadly serve to validate the ESC approach.
Finally, the ESC estimator implies that assignment to a small class in 3rd grade (which has 25\% fewer students on average) increases the probability of graduating from a New York City public high school by $0.69$ percentage points (pp), relative to a sample mean of $51.4\%$. This estimate is one of the first estimates of the causal effect of class size reduction on high-school graduation rates in the U.S.
In contrast, the standard OLS estimator yields significant negative estimates on test scores in later grades and on high school graduation rates. When we control for observable characteristics, the OLS estimates remain negative, while the ESC estimates remain similar to the experimental estimates (Figure \ref{fig_lu_zoom_surr_aux}). These findings demonstrate how our proposed experimental selection correction can detect and adjust for selection biases that are difficult to address with conventional methods in observational data without relying on strong surrogacy assumptions.
In addition to the literature on statistical surrogates, our analysis relates to other studies that have examined similar observation schemes, including \citet{rosenman2018propensity, rosenman2020combining, kallus2020role, mealli2013using} and \citet{imbens2025long}. \citet{rosenman2018propensity} focuses on the problem where assignment is unconfounded in both samples and combining the samples increases precision. \citet{rosenman2020combining} allow for unobserved confounders in the observational sample and consider shrinkage estimators to decrease bias. \citet{kallus2020role} analyze the case where assignment in the combined experimental and observational sample is unconfounded, but not in each sample separately. \citet{kallus2018removing} focus on a case where the same variables (including the primary outcome) are observed in the two samples, but where unconfoundedness is violated in the observational sample and the experimental sample is used to estimate bias. \citet{mealli2013using} focuses on an instrumental variables setting where the presence of multiple outcomes improves estimates.
\cite{bhattacharya2013evaluating} proposes combining experimental and observational data for estimating policy rules in a different setting.
Our approach is conceptually closely related to the Changes-in-Changes (CIC) estimator in \citet{athey2006identification}. For a unit in the observational sample, the control function is essentially the rank of the secondary outcome in the distribution of secondary outcomes in the experimental sample with the same treatment. Under our maintained assumptions here, differences in the estimated effect of the treatment between the experimental and observational sample are attributed to violations of unconfoundedness in the observational data.
The paper is organized as follows. Section 2 analyzes a linear model that captures the intuition underlying our approach. Section 3 presents the identification result for the general case. Section 4 discusses estimation. Section 5 presents the application. Section 6 concludes.
\section{Linear Models}\label{two_examples}
In this section, we introduce our key identifying assumption and a control function estimator in the context of linear models. The linear case simplifies exposition and captures the key ideas that apply in more general models.
\subsection{Setup}
Using the potential outcome set up for observational studies introduced by \citet{rubin1974estimating} (see \citet{imbens2015causal} for a textbook discussion), let the pair of potential outcomes for the primary outcome for unit $i$ be denoted by $Y_{i}^{\rm P}(0)$ and $Y_{i}^{\rm P}(1)$, where the superscript ``${\rm P}$'' stands for ``Primary''. In many applications, $Y_{i}^{\rm P}$ is a long-term outcome; in our application, it is a binary indicator for high school graduation. The treatment received by unit $i$ is $W_{i}\in\{0,1\}$. In our application, $W_i$ is an indicator for small class size in third grade, with $W_i=1$ indicating a small class size and $W_i=0$ indicating a regular class size. There is also a secondary outcome, with the pair of potential outcomes for unit $i$ denoted by $Y_{i}^{\rm S}(0)$ and $Y_{i}^{\rm S}(1)$, where the superscript ``${\rm S}$'' stands for ``Secondary''. In our application, $Y_{i}^{\rm S}$ is a student's end-of-third-grade test score.
We focus in this section on the case where both the primary and secondary outcomes are scalars, but both may be vector-valued ({\it e.g.,} test scores in multiple grades could be used as secondary outcomes).
The realized values for the primary and secondary outcomes are $Y^{\rm P}_i\equiv Y_i^{\rm P}(W_i)$ and $Y^{\rm S}_i\equiv Y_i^{\rm S}(W_i)$. We may also observe pretreatment variables, denoted by $X_i$, that are known not to be affected by the treatment.
We focus on identifying the average treatment effect on the primary outcome,
\begin{equation}\label{taup} \tau^{\rm P}\equiv\mathbb{E}\left[ Y^{\rm P}_i(1)-Y^{\rm P}_i(0)\right],\end{equation}
although other estimands such as the average effect on the treated can be accommodated in our set up as well. The average treatment effect on the secondary outcome, $ \tau^{\rm S}\equiv \mathbb{E}\left[ Y^{\rm S}_i(1)-Y^{\rm S}_i(0)\right],$ is, for the purpose of the current study, not of intrinsic interest.
We have two samples to draw on to estimate $\tau^{\rm P}$, as in the literature on combining datasets, {\it e.g.,} \citet{hotz2005predicting, ridder2007econometrics, pearl2014external}. The first is an observational study that is a random sample from the population of interest. For all units in this observational sample, we observe the quadruple $(W_i,Y^{\rm S}_i,Y^{\rm P}_i,X_i)$.
The second sample is a possibly selective sample from the same population, with random assignment of treatment $W_i$. For all units in this experimental sample, we observe the triple $(W_i,Y^{\rm S}_i,X_i)$, but not the primary outcome $Y^{\rm P}_i$.
Let $G_i\in\{{\rm E},{\rm O}\}$, be an indicator for the sample a unit is drawn from. Then we can conceptualize the combined sample as a random sample of size $N$ from an artificial super-population for which we observe the quintuple $(W_i,G_i,Y^{\rm S}_i,Y^{\rm P}_i\boldsymbol{1}_{G_i={\rm O}},X_i)$, where $\boldsymbol{1}_{G_i={\rm O}}$ is a binary indicator, equal to 1 if $G_i={\rm O}$ and equal to 0 if $G_i={\rm E}$.
\subsection{A Control Function Estimator}
Suppose we have a linear model for the secondary potential outcomes in combination with a constant treatment effect $\tau^{\rm S}$:
\[ Y^{{\rm S}}_i(0)=X_i^\top\gamma^{\rm S}+\alpha_i^{\rm S},\hskip1cm Y^{\rm S}_i(1)=Y^{{\rm S}}_i(0)+\tau^{\rm S}
.\]
This model holds in both the experimental and observational samples. However, the properties of the unobserved component $\alpha^{\rm S}_i$ differ between the two samples. In the experimental sample, randomization guarantees the following conditional independence condition:\footnote{In fact the randomization implies an even stronger condition, $W_i \perp\!\!\!\perp\alpha_i^{\rm S},X_i|G_i={\rm E}$, but we do not need that condition here.}
\[ W_i\ \perp\!\!\!\perp\ \alpha_i^{\rm S}\ \Bigl|\ X_i,G_i={\rm E}.\]
In the observational study, the same conditional independence does not generally hold:
\[ W_i\ \not\!\perp\!\!\!\perp\ \alpha_i^{\rm S}\ \Bigl|\ X_i,G_i={\rm O}.\]
We specify a similar linear model for the primary outcome, but allow the coefficients to be different from those of the model for the secondary outcome:
\[ Y^{{\rm P}}_i(0)=X_i^\top\gamma^{\rm P}+\alpha_i^{\rm P},\hskip1cm Y^{{\rm P}}_i(1)= Y^{{\rm P}}_i(0)+\tau^{\rm P}.\]
Again, the unobserved component might be correlated with the treatment in the observational sample:
\[W_i\ \not\!\perp\!\!\!\perp\ \alpha_i^{\rm P}\ \Bigl|\ X_i,G_i={\rm O},\]
that is, $W_i$ is again endogenous.
To identify the treatment effect on the primary outcome $\tau^{\rm P}$ in the observational sample, we make the following assumption that links the endogeneity problems for the primary and secondary outcomes:
\begin{equation}\label{cf_linear}\alpha^{\rm P}_i=\delta\alpha^{\rm S}_i+\varepsilon_i^{\rm P},
\hskip1cm {\rm with}\ \
W_i\ \perp\!\!\!\perp\ \varepsilon_i^{\rm P}\ \Bigl|\ X_i,\alpha^{\rm S}_i,G_i={\rm O}.\end{equation}
This assumption requires that the component of the residual in the primary outcome that is not explained by the residual in the secondary outcome, $\varepsilon_i^{\rm P}\equiv \alpha_i^{\rm P}-\mathbb{E}[\alpha^{\rm P}_i|\alpha^{\rm S}_i]$, is orthogonal to treatment. The key substantive restriction captured by this condition is that the unobserved confounders that affect the secondary outcome are the same as those that affect the primary outcome. This assumption, which we term \textit{latent unconfoundedness}, is the key to identifying $\tau^{\rm P}$ in the general case below as well.
We now show how this latent unconfoundedness assumption allows us to identify $\tau^{\rm P}$ using a simple control function approach in the linear case. First, we exploit randomization in the experimental sample to estimate $\tau^{\rm S}$ and $\gamma^{\rm S}$ using ordinary least squares regression. Denote these least squares estimates by $\hat\tau^{{\rm S}}$ and $\hat\gamma^{{\rm S}}$.
Next, we estimate the residual $\alpha_i^{\rm S}$ for the units in the observational sample as
\begin{equation}\label{alpha} \hat \alpha^{\rm S}_i=Y^{\rm S}_i-W_i\hat\tau^{{\rm S}}-X_i^\top\hat\gamma^{{\rm S}}.\end{equation}
If the assignment to treatment in the observational sample were random (and assuming the linear model is correct), the population value of these residuals $\alpha^{\rm S}_i$ would be uncorrelated with the treatment indicator in the observational sample.
When treatment assignment is non-random, we can use the association between the secondary outcome residuals $\alpha_i^{\rm S}$ and the treatment to correct for selection bias in the estimating equation for the primary outcome. We do so by including $\alpha_i^{\rm S}$ as a control variable in an ordinary least squares regression of the primary outcome on treatment. To see why this yields a consistent estimate of $\tau^{\rm P}$, observe that we can use the linear representation in (\ref{cf_linear}) to write the primary outcome as:
\begin{equation}\label{control} Y^{{\rm P}}_i=W_i\tau+X_i^\top\gamma+\delta\alpha^{\rm S}_i+\varepsilon^{\rm P}_i,\hskip1cm
{\rm with}\ \
W_i \perp\!\!\!\perp\ \varepsilon_i^{\rm P}\ \Bigl|\ X_i,\alpha^{\rm S}_i,G_i={\rm O}.\end{equation}
Because the error term $\varepsilon_i^{\rm P}$ is orthogonal to treatment in this specification, estimating this equation using OLS yields a consistent estimator for $\tau^{\rm P}$ under our assumptions.
\section{The General Case}
In this section, we generalize the linear example above to accommodate (i) non-linear models and (ii) multiple secondary outcomes.
We are interested in causal estimands defined for the population of interest. Such estimands include simple average treatment effects, but more generally also the average effect of a policy that assigns the treatment to individuals in this population on the basis of covariates ({\it e.g.}, \citet{manski2004statistical, dehejia2005program, hirano2009asymptotics, athey2017efficient, zhou2018offline}). For expositional simplicity, we focus here on average treatment effects. Define
\begin{equation}\label{tau_general}\tau_{g}^{t}\equiv \mathbb{E}\left[\left.Y_{i}^{t}(1)-Y_{i}^{t}(0)\right|G_{i}=g\right],\end{equation}
to be the average effect of the treatment on outcome $t\in\{{\rm S},{\rm P}\}$ for group $g\in\{{\rm O},{\rm E}\}$. The superscripts on the estimands denote the outcome, and subscripts denote the population. The primary estimand we focus on in this paper is the average effect of the treatment on the primary outcome in the observational study population:
\begin{equation}\label{tau}\tau\equiv \tau_{{\rm O}}^{{\rm P}}\equiv \mathbb{E}\left[\left.Y_{i}^{{\rm P}}(1)-Y_{i}^{{\rm P}}(0)\right|G_{i}={\rm O}\right],\end{equation}
where we drop the subscript and superscript to simplify the notation.
\subsection{Three Maintained Assumptions}
There are three key features of our set up. First, we are interested in the population that the units in the observational study were drawn from. That is, the observational study has external validity.
\begin{assumption}{\sc (External Validity of the Observational Study)} The observational sample is a random sample of the population of interest.
\label{assumption:external_observational}
\end{assumption} \noindent This can be thought of as simply defining the estimand in terms of the population distribution underlying the observational sample.
Second, we maintain throughout the paper the assumption that
the treatment in the experimental sample is unconfounded.
\begin{assumption}\label{assumption:random}{\sc (Internal Validity of the Experimental Sample)}
For $w=0,1$,
\begin{equation}\label{random} W_i\ \perp\!\!\!\perp\ \Bigl(Y_{i}^{{\rm P}}(w),Y_{i}^{{\rm S}}(w)\Bigr)\ \Bigr| \ X_i,G_i={\rm E}.\end{equation}
\label{assumption:internal_experimental}\end{assumption}
Although internal validity of the experimental sample is guaranteed by design,
external validity of the experimental study does not follow.
We assume that conditional on the pretreatment variables we have external validity (\citet{hotz2005predicting}):
\begin{assumption}\label{assumption:conditional}{\sc (Conditional External Validity)} The experimental study has conditional external validity if
\begin{equation}\label{eq:cond_ext} G_i\ \perp\!\!\!\perp\ \Bigl(Y_{i}^{{\rm P}}(0),Y_{i}^{{\rm P}}(1),
Y_{i}^{{\rm S}}(0),Y_{i}^{{\rm S}}(1)\Bigr)\ \Bigr|\ X_i.\end{equation}
\end{assumption}
This assumption implies that if we find systematic differences between in differences in average outcomes by treatment status conditional on covariates between the experimental and observational sample, these differences must arise from violations of unconfoundedness for the observational sample.
The first result is that these three maintained assumptions are in general not sufficient for point-identification of the average effect of interest. Of course this does not mean that these assumptions do not have any identifying power. They do in fact imply non-trivial identified sets in the spirit of the work by (\citet{manski_bounds}).
\begin{lemma}\label{lemma1}
The combination of Assumptions \ref{assumption:external_observational}-\ref{assumption:conditional} is not sufficient for point-identification of $\tau^{\rm P}$.
\end{lemma}
The proof for this result is given in the appendix.
Next, let us briefly mention a common assumption that we do {\it not} wish to make in this context. Specifically, we consider the assumption that assignment in the observational study is unconfounded.
For $w=0,1$,
\begin{equation}\label{unc}W_{i}\ \perp\!\!\!\perp\ \Bigl(Y_{i}^{{\rm S}}(w),Y_{i}^{{\rm P}}(w)\Bigr)\ \Bigl|\ X_{i},G_i={\rm O},\end{equation}
This assumption is made, for example, in \citet{rosenman2018propensity}.
This assumption is sufficient for identification of $\tau$, but it is stronger than necessary. Intuitively it implies that we do not need the experimental sample for identification because under unconfoundedness the observational sample is sufficient for identification of the average treatment effect. However, the experimental sample may still be useful for precision.
\subsection{Latent Unconfoundedness}
Suppose that we reject the combination of Assumptions \ref{assumption:random}-\ref{assumption:conditional} and unconfoundedness (\ref{unc}). If we maintain unconfoundedness in the experimental sample (Assumption \ref{assumption:random}), it must be that either conditional external validity in the experimental study (Assumption \ref{assumption:conditional}), or unconfoundedness in the observational study (\ref{unc}) must be violated.
In many cases we may wish to maintain conditional external validity and interpret the finding that the combination
does not hold as evidence that unconfoundedness in (\ref{unc}{) does not hold for the observational study.
The fundamental idea behind our approach, although not the implementation, can be seen as related to that in a Difference-In-Differences (\citet{cardmariel, cardkrueger1, angrist2008mostly}) set up where the initial (pre-treatment) differences between a treatment and control group are used to adjust post-treatment differences between the treatment and control group. More specifically, it relates to the Changes-In-Changes approach in \citet{athey2006identification} where functional form assumptions are avoided. Here initial differences in treatment effects between an experimental and observational study are used to adjust subsequent treatment effects for the observational study.
They key additional assumption that links the biases between adjusted comparisons for the primary and secondary outcomes, is the following.
\begin{assumption}\label{assumption:new}{\sc (Latent Unconfoundedness)}\\
For $w\in\{0,1\}$,
\begin{equation}\label{new}
W_i\ \perp\!\!\!\perp\ Y_{i}^{{\rm P}}(w)\ \Bigl|\ X_{i},Y^{\rm S}_i(w),G_i={\rm O}
.\end{equation}
\end{assumption}
This assumption is both novel as well as critical in the current discussion, so we offer some remarks.
\begin{remark}
Compared to a regular unconfoundedness assumption, we add the variable $Y^{\rm S}_i(w)$ to the conditioning set. At first this may appear to be an innocuous addition. However, following the standard approach to exploiting unconfoundedness assumptions, we see that this is not the case. Typically we use an unconfoundedness assumption to create subpopulations defined by the conditioning variables, and then compare treated and control units within those subpopulations. To be specific, suppose we wish to estimate $\mathbb{E}[Y^{\rm P}_i(1)|G_i={\rm O}]$. We would first estimate the conditional expectation
$\mu(y^{\rm S},x)=\mathbb{E}[Y^{\rm P}_i(1)|Y_i^{\rm S}(1)=y^{\rm S},W_i=1,X_i=x,G_i={\rm O}]$. Then, in the second step, we would average this over the marginal distribution of $(Y_i^{\rm S}(1),X_i)$ in the observational sample. However, in the observational sample we only see draws from the conditional distribution of $(Y_i^{\rm S}(1),X_i)$ given $W_i=1$, and this is {\it not} the same distribution because of the failure of unconfoundedness in the observational sample. To address this, we need to exploit the presence of the experimental sample.
\end{remark}\begin{remark}
The precise version of the unconfoundedness assumption here is slightly different from than the (stronger) unconfoundedness assumption in, say, \citet{rosenbaum1983central} where it is assumed that $W_i$ is independent of the full set of $Y_i^{\rm P}(0),Y^{\rm P}_i(1))$. It is what is referred to in \citet{imbens2000} as ``weak unconfoundedness.''
\end{remark}
To highlight the link to the control function literature (\citet{heckman1979sample, heckman1985alternative, imbens2009identification, wooldridge2010econometric, athey2006identification, kline2019heckits, mogstad2018using, mogstad2018identification, wooldridge2015control}), let us model the primary and secondary potential outcomes as
\[Y^{\rm P}_i(w)=h^{\rm P}(w,\nu_i,X_i),\hskip1cm {\rm and}\ \ Y^{\rm S}_i(w)=h^{\rm S}(w,\eta_i,X_i)
,\]
with the function $h^{\rm S}(w,\eta,x)$ strictly monotone in $\eta$.
In the context of this model we can write the latent unconfoundedness assumption as
\[ W_i\ \perp\!\!\!\perp\ \nu_i\ \Bigl|\ X_{i},\eta_i,G_i={\rm O}.\]
Although it is not generally true that $W_i \perp\!\!\!\perp \nu_i| X_{i},G_i={\rm O}$ (without conditioning on $\eta_i$), adding $\eta_i$ to the conditioning set restores the exogeneity of $W_i$ in the observational sample.
It is useful to contrast this with a control function in a nonparametric instrumental variables setting ({\it e.g.,} \citet{imbens2009identification}), where the two models are
\[Y^{\rm P}_i(w)=h^{\rm P}(w,\nu_i,X_i),\hskip1cm {\rm and}\ \ W_i(z)=r(z,\eta_i,X_i)
,\]
with $r(z,\eta,x)$ strictly monotone in $\eta$.
The key assumption here is that
\[ W_i\ \perp\!\!\!\perp\ \nu_i\ \Bigl|\ X_{i},\eta_i.\]
The model relating the outcome of interest and the endogenous regressor is essentially the same in the two settings, $Y^{\rm P}_i(w)=h^{\rm P}(w,\nu_i,X_i)$. In both cases we address the endogeneity by conditioning on an additional variable, the control variable $\eta_i$. This control variable is estimated using an auxiliary model. This auxilliary model differs between the set up in the current paper and the instrumental variables setting \citet{imbens2009identification}.
In the Imbens-Newey nonparametric instrumental variables setting we model the relation between the endogenous regressor and an additional variable, the instrument, and deriving the control variable from that relation.
In the current setting we model the relation between the secondary outcome and the endogenous regressor and deriving the control variable from that relation. In both cases the auxiliary model has a strict monotonicity assumption. This comparison shows one of the limitations of the approach: the unobserved confounder $\eta_i$ cannot have a dimension higher than that of the secondary outcome.
Formally, adding Assumption \ref{assumption:new} (latent unconfoundedness) to Assumptions \ref{assumption:external_observational}-\ref{assumption:conditional} allows us to point-identify the average effect of interest. The following theorem states our main identification result.
\begin{theorem}\label{theorem_main}
Suppose that Assumptions \ref{assumption:external_observational}-\ref{assumption:new} hold, so that the experimental study is unconfounded and has conditional external validity, and the observational study has latent unconfoundedness. Then
the average effect of the treatment on the primary outcome in the observational study is point-identified.
\end{theorem}
\subsection{Missing At Random}
There is an interesting connection between Assumptions \ref{assumption:external_observational}-\ref{assumption:new} and the Missing At Random (MAR) assumption in the missing data literature (\citet{rubin1976inference, little2019statistical, rubin2004multiple}).
\begin{lemma}
Suppose that Assumptions \ref{assumption:external_observational}-\ref{assumption:new} hold. Then:
\begin{equation}\label{mar}G_i\ \perp\!\!\!\perp\ Y_{i}^{{\rm P}}\ \Bigl|\ W_{i},X_{i},Y_{i}^{{\rm S}}.\end{equation}
\end{lemma}
Because $G_i={\rm E}$ is equivalent to an indicator that $Y^{\rm P}_i$ missing, and because $W_i$, $X_i$, and $Y^{\rm S}_i$ are observed for all individuals in the sample, the conditional independence in (\ref{mar}) is equivalent to a MAR assumption. The result does not go the other way around. The MAR assumption by itself has no testable implications, but the combination of Assumptions \ref{assumption:external_observational}-\ref{assumption:conditional} and \ref{assumption:new} does imply some inequality restrictions on the joint distribution of the observed variables.
\citet{kallus2020role} starts with a MAR assumption, and uses that in combination with an unconfoundedness assumption on the full sample to identify the average effect of the treatment for the full sample.
\section{Estimation and Inference}
In this section, we discuss estimation and inference.
There are multiple approaches here, some of which we discussed in the examples in Section \ref{two_examples}. These strategies include imputation, weighting, control function methods, and influence-function based methods. Because the model is just-identified, all four of these methods are first-order equivalent, although they will have different finite sample properties, see
\citet{newey1994asymptotic, chen2018overidentification} for a general discussion.
We focus here on the control function approach that is special to this setting with observational and experimental data. In the appendix, we discuss the imputation, weighting, and influence function approaches, which closely resemble their equivalents in standard unconfoundedness settings.
In the control function approach, we directly estimate the unobserved confounder. We then estimate the average treatment effect in the observational sample adjusting for both the observed covariates and the estimated confounder. The procedure consists of three steps.
In the first step, we estimate the conditional cumulative distribution function of the secondary outcome, conditional on the treatment and pre-treatment variables, in the experimental sample:
\[ F_{Y^{\rm S}|W,X}(y^{\rm S}|w,x)\equiv {\rm pr}(Y^{\rm S}_i\leq y^{\rm S}|W_i=w,X_i=x,G_i={\rm E}).\]
Note that if the secondary outcome is a vector, this is a vector of conditional cumulative distribution functions, demonstrating that multiple secondary outcomes can weaken the identifying assumptions.
In the second step, we calculate for all units in the observational sample the control variable as
\[ \eta_i= F_{Y^{\rm S}|W,X}(Y_i^{\rm S}|W_i,X_i).\]
In the third step, we estimate the adjusted difference
\[ \mathbb{E}\left[\left.
\mathbb{E}\left[\left.Y^{\rm P}_i\right|W_i=1,X_i,G_i={\rm O}\right]
-
\mathbb{E}\left[\left.Y^{\rm P}_i\right|W_i=0,X_i,G_i={\rm O}\right]
\right| G_i={\rm O}\right],\]
which by the assumptions in Theorem \ref{theorem_main} is equal to the average causal effect $\tau$. Here we can use any of the conventional methods for estimating average treatment effects under unconfoundedness for the observational data (including matching, regression, inverse propensity score weighting, augmented inverse propensity score weighting, or doubly robust methods), where we use the combination of the pretreatment variables $X_i$ and the estimated control function $\eta_i$ as the variables to be adjusted for. For example, using a imputation/regression approach, one would estimate the conditional mean of the primary outcome in the observational sample given treatment status, control variable, and pre-treatment variables:
\[\gamma(w,h,x)\equiv \mathbb{E}\left[\left. Y^{\rm P}_i\right|W_i=w, \eta_i=h,X_i=x,G_i={\rm O}\right].\]
These estimated conditional means would then be use to estimate the average treatment effect $\tau$ as
\[ \hat\tau^{\rm cf}=\frac{1}{N^{\rm O}_1}\sum_{i:G_i={\rm O}} W_i \hat\gamma(1,\hat\eta_i,X_i)-
\frac{1}{N^{\rm O}_0}\sum_{i:G_i={\rm O}} (1-W_i) \hat\gamma(1,\hat\eta_i,X_i),\]
where $N^g_w=\sum_{i=1}^N {\bf 1}_{W_i=w,G_i=g}.$
Under standard conditions ({\it e.g., } \citet{newey1994asymptotic, chen2018overidentification}) this estimator will be semiparametrically efficient and asymptotically linear and normally distributed.
From \citet{newey1994asymptotic} it follows that the control function estimator is first order equivalent to the imputation, weighting, and influence function estimators, and so one can use the semiparametric efficiency bound for variance estimation. Alternatively, one can use the regular bootstrap.
\section{Application: Effects of Class Size on Graduation}
In this section, we evaluate the performance of our approach by estimating the long-term impacts of reducing class sizes in elementary school. Many experimental and quasi-experimental studies have analyzed the impacts of educational inputs – such as class size, teacher quality, and resources – on test scores \citep{krueger1999experimental,kane2008estimating,biasi2025works}. Meanwhile, observational data with information on educational inputs as well as long-term outcomes such as high school graduation rates from school districts’ administrative records have become widely available. We combine these two sets of information to estimate the effects of class size on high school graduation rates.\footnote{A few studies have linked experimental data to administrative data from tax records and other sources to measure impacts on outcomes such as college attendance rates and earnings (see e.g., Chetty et al. 2011 STAR, Dynarski et al STAR, Fredrikkson et al. QJE on class size effects). However, such linkages remain challenging and relatively rare. Our objective here is to show how one can make progress in identifying treatment effects of interest even when direct measurement of primary outcomes in the experimental sample is infeasible.} We first describe the data we use and then present results.
\subsection{Data}
We combine information from two datasets: experimental data from Tennessee STAR and observational data from the New York City public school district.
\textit{Experimental Sample: Tennessee STAR}. The STAR experiment was conducted at 79 low-income public schools in Tennessee between 1985-89. In the 1985-86 school year, 6,323 kindergarten students in participating schools were randomly assigned to a small (target size 13-17 students) or regular-sized (20-25 students) class within their schools. An additional 5,248 children joined the 1985-86 entry cohort at the participating schools after kindergarten in grades 1-3. These new entrants were also randomly assigned to small vs. large classrooms within school upon entry. Students were intended to remain in the same class type (small vs. large) through 3rd grade, at which point all students returned to regular class sizes.
In each year from grades 3-8, STAR students were administered standardized tests that measure performance in math and reading. We standardize the average of math and reading scores to have mean 0 and standard deviation 1 within each grade among students in the STAR sample. We also observe information on students’ race and ethnicity, sex, and eligibility for free or reduced-price lunch (an indicator for having low-income parents). For further information on the STAR experiment, see \citet{word1990state} , \citet{krueger1999experimental}, and \citet{chetty2011does}.
\textit{Observational Sample: New York City}. We obtain observational information from the administrative records of the New York City public school district for 1.76 million children in grades 3-8 between the 1991-2009 school years. Starting from the raw data, we impose the same sample restrictions as in \citet{chetty2014onemeasuring} – such as excluding special education classrooms and classrooms with less than 10 or more than 50 students – and additionally limit the sample to students for whom we observe test scores throughout grades 3-8. For comparability to the STAR treatment of dichotomous assignment to small vs. large classes, we define a “small” class in NYC as one with 26 (the sample median) or fewer students in third grade.
We observe math and reading test scores at the end of grades 3-8, which we standardize to have mean 0 and standard deviation within each grade in the NYC sample.\footnote{Chetty et al. (2014) show that the within-grade variation in achievement in the NYC school district is comparable to the within-grade variation in other urban school districts nationwide, and hence is likely comparable to that in the STAR population, which exhibits broadly similar socioeconomic characteristics.} Critically, unlike in the STAR sample, we also observe an indicator for graduating from a NYC public high school by 2016, which we view as the primary outcome of interest.\footnote{We can only observe whether students graduated from a high school in the New York City public school district. 27\% of students in our sample leave the NYC school district before the end of high school; we include these students in our analysis and code them as not graduating from an NYC high school. The estimated effects of class size reduction on graduation rates are larger in schools where fewer students leave the district, suggesting that this missing data issue leads us to understate the overall impact of class size reduction on graduating from any high school.} We also observe information on students' race and ethnicity, sex, and (after the 1999 school year) eligibility for free or reduced-price lunch. For further information on the New York City data, see \cite{chetty2014onemeasuring} and \citet{mariano2024effects}.
\textit{Summary Statistics.} Appendix Table \ref{tab:merged_summary_statistics} presents summary statistics for the two samples. Although they are from different time periods and geographic settings, the two samples overlap on key student characteristics. Both districts serve primarily low-income students, with 61\% of students in the STAR sample and 81\% of the students in the NYC sample eligible for free or reduced-price lunches. Approximately one-third of the students are Black in both datasets, while the New York City sample has a significantly larger share of Hispanic students than the STAR sample. On average, there are 7.0 fewer students in third grade classrooms defined as “small” in the New York City data and 6.7 fewer students in the classrooms of students assigned to small classes in the STAR sample. 51\% of students in New York City public schools graduated from high school, consistent with official statistics \citep{nysed_downloads}.
\subsection{Results}
We begin with OLS regressions of 3rd grade test scores on an indicator for being assigned to a small class in the STAR and NYC datasets.\footnote{Throughout our analysis of the STAR data, we use initial assignment to small class (rather than actual realized class size) as the independent variable, thereby reporting intent-to-treat estimates. Compliance with treatment assignment was imperfect because principals had to re-balance classes on other dimensions, such as gender composition. The ITT estimates provide the appropriate scaling for comparison to the observational NYC sample because the difference in average class size between those initially assigned to small vs. large classes in STAR of 6.7 students is comparable to that in the New York City data.} We include school fixed effects in all regressions run in the STAR sample because randomization was conducted within schools among children who entered in a given birth cohort. We analogously include school and birth cohort fixed effects in the NYC sample to isolate within-school and cohort variation in class size.
Table 1 (in the introduction) reports estimates from these regressions. In the STAR sample (Column 1), small class assignment in third grade increases end-of-third-grade test scores by 0.19 SD (se= 0.04).\footnote{Students who entered STAR schools before 3rd grade and were assigned to small classes in 3rd grade were assigned to small classes in earlier grades as well. We find that assignment to a small class has similar effects of end-of-3rd-grade test scores for those who entered STAR schools in 3rd grade (and thus were treated for only one year) as for those who entered in earlier grades. This is a consequence of the rapid fade-out of treatment effects on subsequent test scores documented in Figure 2. We therefore interpret the treatment effect on test scores as the causal effect of being assigned to a small class in third grade when we construct an analogous observational estimate in the NYC sample.} In the NYC sample, the corresponding OLS estimate is -0.12 SD (se = 0.01). The difference between these estimates implies that the observational estimates are confounded under our maintained external validity assumption (Assumption \ref{assumption:external_observational}).
Next, we implement the ESC estimator by estimating equation \eqref{control}. We first calculate the difference between actual 3rd grade test scores and predicted test scores based on the student’s class size ($\alpha^{\rm S}_i= Y^{\rm S}_i - \tau^{\rm S} W_i$) in the NYC sample. We then replicate the OLS specification in Column 2, additionally controlling for the residuals $\alpha^{\rm S}_i$. The ESC correction yields an estimated treatment effect of class size on third grade test scores in the NYC sample of 0.19 SD (se = 0.04), which coincides with the experimental estimate in the STAR sample by construction (Column 3 of Table 1).\footnote{We estimate standard errors for this and all other ESC estimates reported below using a bootstrap procedure, where we resample the student‐level observations with replacement $B=1{,}000$ times, re‐estimate both the first‐stage class‐size effect $\hat\tau^{\rm S}$ (to form residuals $\alpha_i^{\rm S}$) and the second‐stage ESC OLS controlling for $\alpha_i^{\rm S}$ in each replication, and compute standard errors as the empirical standard deviation of the resulting bootstrap estimates.}
We next use the ESC estimator to estimate treatment effects on subsequent outcomes. Figure \ref{fig_lu_main} plots treatment effects on test scores in grades 3-8. The STAR experimental estimates (shown in black squares) are positive in all grades, while the OLS estimates in the NYC data (orange triangles) are all negative. The ESC estimates in grades 4-8 are all positive and very similar in magnitude and temporal pattern to the STAR estimates. Notably, the ESC estimates capture the well-known “fadeout” pattern in the STAR estimates – where the effects of interventions in early grades on test scores diminish in later grades. The close correspondence between the ESC estimates and the experimental estimates for the holdout outcomes ($Y^H_i$) of test scores in grades 4-8 supports the latent unconfoundedness assumption and demonstrates the ability of our approach to adjust for selection.
Finally, in the right panel of Figure \ref{fig_lu_main}, we turn to the primary outcome of interest – high school graduation – which we observe in the observational but not experimental sample. The ESC estimator implies that assignment to a small third grade class increases the probability of graduating from a NYC high school by 0.69 percentage points (Column 3 of Table 1). Small classes have 7 fewer students on average relative to a sample mean of 28 students; hence, a 25\% reduction in class size in third grade increases high school graduation rates by 0.69 pp.
In contrast, the OLS estimator yields a negative association between small class assignment and high school graduation rates in the observational sample. Importantly, adjusting for selection on observables by controlling flexibly for key demographic covariates in the observational sample – the interaction of indicators for gender, race and ethnicity, eligibility for free or reduced-price lunch, and cohort – has little impact on the OLS estimates on test scores and graduation rates (Figure \ref{fig_lu_zoom_surr_aux}, Appendix Table \ref{tab:hs_graduation_summary}). This result demonstrates that the experimental selection correction can adjust for selection on dimensions that are typically unobserved in standard administrative datasets.
\begin{figure}[H]
\centering
\caption{Effect of Controlling for Observables on Treatment Effect Estimates}
\includegraphics[width=1\linewidth]{fig_naiveols_sc_auxcontrols.pdf}
\vspace{0.2cm}
\label{fig_lu_zoom_surr_aux}
\caption*{\footnotesize
\textit{Notes:} This figure shows how controlling for demographic observables influences the estimated effect of assignment to a small 3rd-grade class on test scores (left panel) and high school graduation rates (right panel). “ESC” refers to the Experimental Selection Correction estimate, and “OLS” denotes the Observational OLS estimate in the NYC sample, constructed as described in the notes to Figure 2. All specifications include school and cohort fixed effects. Estimates labeled “With Demog. Controls” additionally control for the interaction of indicators for gender, race and ethnicity, eligibility for free or reduced-price lunch, and cohort, as well as an indicator for missing the free lunch variable (in the NYC data). Vertical bars represent 95\% confidence intervals.}
\end{figure}
\textit{Robustness.} We find very similar estimated impacts on high school graduation rates when focusing on specific demographic groups ({\it e.g.}, by race or sex), as shown in Figure \ref{fig_lu_int_robust}. These findings allay the concern that differences in the demographic distribution between the STAR and NYC samples may lead to violations of the external validity assumption. We also find that the estimated impacts on high school graduation remain similar when we correct for selection using all test scores from grades 3-8 instead of just 3rd grade scores (Figure \ref{fig_lu_hetero_robust}). These findings are consistent with the finding that treatment effects on test scores in grades 4-8 are very similar in the NYC and STAR data once we adjust for selection using 3rd grade test scores (Figure \ref{fig_lu_main}).
\begin{figure}[H]
\centering
\caption{Heterogeneity of Treatment Effect Estimates on HS Graduation Across Subgroups}
\includegraphics[width=0.95\linewidth]{fig_hetero_subgroup.pdf}
\label{fig_lu_int_robust}
\caption*{\footnotesize
\textit{Notes:} This figure presents subgroup-specific estimates of the effect of being assigned to a small 3rd-grade class on high school (HS) graduation rates. Each point corresponds to a point estimate obtained using either the observational OLS or Experimental Selection Correction (ESC) estimator in the NYC data, constructed as described in the notes to Figure 2. All specifications include school and cohort fixed effects. Estimates are expressed as percentage-point changes in HS graduation rates. Vertical lines represent 95\% confidence intervals.
}
\end{figure}
\begin{figure}[H]
\centering
\caption{Robustness of ESC Estimates to Choice of Intermediate Outcomes}
\includegraphics[width=0.7\linewidth]{fig_robust_inter.pdf}
\label{fig_lu_hetero_robust}
\caption*{\footnotesize
\textit{Notes:} This figure illustrates how the estimated effect of assignment to a small 3rd-grade class on high school graduation rates varies with the intermediate outcomes used in the experimental selection correction procedure. Green points with vertical bars plot ESC estimates accompanied by 95\% confidence intervals. The first estimate replicates the ESC estimate reported in Column 3 of Table 1, using only 3rd grade test scores for selection correction. The remaining estimates use additional test scores from grades 4-8 as intermediate outcomes in the selection correction procedure. The orange line shows the observational OLS estimate in the NYC sample from Column 2 of Table 1 (which does not use intermediate outcomes and hence is constant across the figure). The dashed orange line shows a 95\% confidence interval for the OLS estimate.}
\end{figure}
\textit{Comparison to Surrogate Estimates.} In Figure \ref{fig_lu_zoom_surr_sc}, we compare the ESC estimates to estimates from a surrogate index approach \citep{athey2019surrogate}. To construct the surrogate-based estimates, we multiply treatment effects on third grade test scores in the STAR sample by coefficients from OLS regressions of the outcome of interest on third grade scores in the NYC sample (including school and cohort fixed effects). The surrogate estimates on test scores in grades 4-8 are all higher than the experimental estimates (though not significantly so), while the ESC estimates match the experimental estimates more closely. This pattern is consistent with the education literature on fadeout, which finds that treatment effects of early interventions on test scores persist less than one would expect based on the serial correlation of test scores across grades, violating the assumption that test scores in earlier grades provide surrogates for later outcomes. Accordingly, the ESC estimate yields an estimated treatment effect on high school graduation that is about half as large as the surrogate-based estimate.
\begin{figure}[H]
\centering
\caption{Comparison of Surrogate and ESC Estimates}
\includegraphics[width=1\linewidth]{fig_surr_comp.pdf}
\label{fig_lu_zoom_surr_sc}
\caption*{\footnotesize
\textit{Notes:} This figure compares the ESC estimates reported in Figure 2 to estimates that use third grade test scores as a surrogate for subsequent outcomes. The surrogate estimates are constructed by multiplying treatment effects on third grade test scores in the STAR sample by coefficients from OLS regressions of the outcome of interest (test scores in later grades or HS graduation) on third grade scores in the NYC sample. The ESC and experimental estimates reproduce the series plotted in Figure 2. See notes to Figure 2 for further details.
}
\end{figure}
\begin{comment}
\begin{table}[tbh]
\caption{}
\label{tab:}
\centering
\vspace{1em}
\begin{tabular}{lccccc}
\toprule
{} & Secondary & Secondary & Primary & Primary & Imputed Primary \\
\midrule\\
$W_i$ & $0.157$\textsuperscript{} & $-0.048$\textsuperscript{} & $-0.074$\textsuperscript{} & $0.061$\textsuperscript{} & $0.061$\textsuperscript{} \\
& ($0.028$\textsuperscript{})\textsuperscript{} & ($0.002$\textsuperscript{})\textsuperscript{} & ($0.003$\textsuperscript{})\textsuperscript{} & ($0.018$\textsuperscript{})\textsuperscript{} & ($0.018$\textsuperscript{})\textsuperscript{} \\ \\
$\hat\alpha_i^S$ & & & & $0.640$\textsuperscript{} & \\
& & & & ($0.001$\textsuperscript{})\textsuperscript{} & \\ \\
$N$ & $6,027$\textsuperscript{} & $1,131,339$\textsuperscript{} & $498,597$\textsuperscript{} & $498,597$\textsuperscript{} & $6,027$\textsuperscript{} \\
$R^2$ & $0.130$\textsuperscript{} & $0.060$\textsuperscript{} & $0.040$\textsuperscript{} & $0.420$\textsuperscript{} & $0.170$\textsuperscript{} \\
Sample & Experimental & Observational & Observational & Observational & Experimental \\
Covariates & Yes & Yes & Yes & Yes & Yes \\
\bottomrule
\end{tabular}
\end{table}
\end{comment}
\section{Conclusion}
This paper has proposed a new method of combining experimental and observational data to improve causal inference about a primary outcome of interest. We leverage the internal validity of the experimental data to implement a selection correction based on secondary outcomes when estimating treatment effects on the primary outcome in the observational dataset. Our estimator relies on a key new assumption that we term latent unconfoundedness, which requires that the unobserved confounders that affect the primary and secondary outcomes are the same. Our approach strictly weakens the assumptions underlying the popular approach of using intermediate outcomes as surrogates, yielding more credible estimates of causal effects when both the treatment and primary outcome are observed in the observational dataset.
We apply our Experimental Selection Correction estimator to estimate the effects of 3\textsuperscript{rd} grade class size on test scores in later grades and high school graduation rates. In observational data, we find wrong-signed estimates that are likely biased by unobserved selection. The ESC estimator yields estimates that match holdout experimental estimates for test scores in later grades and provides one of the first estimates of the causal effects of class sizes on high school graduation rates in the U.S. -- showing that a 25\% reduction in class size in third grade increases high school graduation rates by 0.7 pp.
As observational data become more widely available, it would be valuable to build on the ideas proposed here by developing approaches to using experiments to correct for selection in observational data. Recent econometric advances have extended the framework we propose here in both identification and estimation—e.g., \citet{meza2024,park2024bracketing, park2024informativeness,imbens2025long}—but there remains substantial scope for further development. In particular, future work could seek to characterize and weaken the latent unconfoundedness condition, potentially by leveraging multiple intermediate variables or partial identification strategies.\footnote{For example, the latent unconfoundedness assumption we use requires that all variation in students’ test scores arises from unobservables that affect graduation rates as well, ruling out shocks that may affect test performance but not later outcomes such as illness or noise on the day of the test. In practice, test-retest reliability tends to be very high (exceeding 0.8), so such noise is likely minimal in our application. But in other settings, noise in the intermediate outcome may be more substantial; in such cases, it may be possible to use instrumental variables approaches to adjust for such noise.} Empirically, it would be useful to characterize settings where latent unconfoundedness is a good approximation using validation studies in order to guide future applications.
\newpage
\clearpage
\small\parindent 0cm\setcounter{equation}{0}
\renewcommand{\arabic{section}.
\arabic{equation}}{A.\arabic{equation}}
\setcounter{lemma}{0}\renewcommand{A.\arabic{lemma}}{A.\arabic{lemma}}
\setcounter{theorem}{0}\renewcommand{A.\arabic{theorem}}{A.\arabic{theorem}}
\setcounter{table}{0}
\setcounter{figure}{0}