Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
109,024 characters · 34 sections · 86 citation commands
Assessing Omitted Variable Bias when the Controls are Endogenous
JEL classification: C18; C21; C51
Keywords: Linear Regression, Sensitivity Analysis, Treatment Effects, Unconfoundedness
\onehalfspacing
For many years, researchers' best approach to dealing with the threat of omitted variables was to either sign the bias, or to use a different identification strategy altogether. These include, for example, measurement error models using proxies, instrumental variable models, or methods for combining datasets where some of the omitted variables are observed. But oftentimes no viable alternative identification strategies are available, and the direction of the bias works against them---meaning that plausible omitted variables could explain away their results. In response to this challenge, the literature developed a variety of results to quantify the impact of omitted variables. A particularly important breakthrough was the paper by AltonjiElderTaber2005, and later extended by Oster2019, who showed how to measure the impact of omitted variables by comparing the magnitude of selection on observed variables with the magnitude of selection on unobserved variables. The tools developed in these papers are now widely used in empirical economics.
However, the recent statistics and econometrics literature has discussed several limitations with those early, first generation methods. This includes DeLucaMagnusPeracchi2019, CinelliHazlett2020, Basu2022, MastenPoirier2025, and a companion paper of ours, DiegertMastenPoirier2025.\footnote{See section 5 of MastenPoirier2025 for a detailed review and explanation of all of these limitations.} In particular, in our companion paper DiegertMastenPoirier2025, we show that previous methods can lead to incorrect conclusions about robustness when the controls are endogenous, meaning that they are correlated with the omitted variables of concern. For example, consider a classic regression of wages on education and controls like parental education. Typically we are worried that the coefficient on education in this regression is a biased measure of the returns to schooling because unobserved ability is omitted. In this example, the controls would be exogenous if unobserved ability is uncorrelated with parents' education, along with all other included controls. Such exogeneity assumptions on the control variables are usually thought to be very strong and implausible. For example, AngristPischke2017 discuss how
Put differently: We usually do not expect the included control variables to be uncorrelated with the omitted variables; instead we merely hope that the treatment variable is uncorrelated with the omitted variables after adjusting for the included controls. These control variables are therefore usually thought to be endogenous.
In this paper, we develop a method for assessing sensitivity to omitted variables that allows the controls to be endogenous. Our approach uses a new set of sensitivity parameters that have several important features:
No other approaches in the literature have all of these features.
In section (ref) we describe the baseline model. The parameter of interest is $\beta_\text{long}$, the coefficient on a treatment variable $X$ in a long OLS regression of an outcome variable $Y$ on a constant, treatment $X$, the observed covariates $W_1$, and some unobserved covariates $W_2$. In section (ref) we discuss causal models which allow us to interpret this parameter causally, based on three different identification strategies: unconfoundedness, difference-in-differences, and instrumental variables. Since $W_2$ is unobserved, we cannot compute the long regression of $Y$ on $(1,X,W_1,W_2)$ in the data. Instead, we can only compute the medium regression of $Y$ on $(1,X,W_1)$. We begin by considering a baseline model with a “no selection on unobservables” assumption, which implies that the coefficients on $X$ in the long and medium regressions are the same. Importantly, this baseline model does not assume the controls $W_1$ are exogenous.
In section (ref) we develop new methods for assessing sensitivity to omitted variables that (a) do not rely on the exogenous controls assumption, (b) satisfy the consistency and monotonicity-in-selection criteria that we develop in our companion paper DiegertMastenPoirier2025, and (c) still allow researchers to compare the magnitude of selection on observables with the magnitude of selection on unobservables. Our identification results can be used with either one or two sensitivity parameters. The first sensitivity parameter compares the relative magnitude of the coefficients on the observed and unobserved covariates in a treatment selection equation. This parameter thus measures the magnitude of selection on unobservables by comparing it with the magnitude of selection on observables. The second sensitivity parameter compares the relative magnitude of the coefficients on the observed and unobserved covariates in the outcome equation. In section (ref), we extend our results to allow researchers to also restrict the magnitude of control endogeneity.
We provide two main identification results. Our first result (Theorem (ref)) characterizes the identified set for $\beta_\text{long}$, the coefficient on treatment in the long regression of the outcome on the treatment and the observed and unobserved covariates. This theorem only requires that researchers make an assumption about a single sensitivity parameter---the relative magnitudes of selection on observables and unobservables. We provide a closed form, analytical expression for the identified set, which makes this result easy to use in practice. Using this result, we show how to do breakdown analysis: To find the largest magnitude of selection on unobservables relative to observables needed to overturn a specific baseline finding. This value, called a breakdown point, can be used to measure the robustness of one's baseline results. We provide a simple expression for the breakdown point and recommend that researchers report estimates of it along with their baseline estimates. This estimated breakdown point provides a scalar summary of a study's robustness to selection on unobservables while allowing for arbitrarily endogenous controls.
If researchers are willing to restrict the impact of unobservables on outcomes, then they can obtain results that are more robust to selection on unobservables. In this case, the identified set is more difficult to characterize analytically (see Theorem (ref) in the appendix). However, our second main identification result (Theorem (ref)) shows that we can nonetheless easily numerically compute breakdown points. We generalize this result in section (ref) to also allow restrictions on the endogeneity of the controls.
In section (ref) we show how to use our results in empirical practice. We use data from BFG2020 who studied the effect of historical American frontier life on modern cultural beliefs. Specifically, they test a well known conjecture that living on the American frontier cultivated individualism and antipathy to government intervention. Among other analyses, they use Oster's (2019) method to argue that omitted variables do not affect their conclusions. Using our results, we obtain richer conclusions about robustness. In particular, when allowing for endogenous controls, we find that effects obtained from questionnaire based outcomes are often sensitive to omitted variables while the effects from election and property tax outcomes remain robust.
We conclude this section with a brief review of the literature. We focus on two literatures: The literature on endogenous controls and the literature on sensitivity analysis in linear or parametric models.
The idea that the treatment variable and the control variables should be treated asymmetrically in the assumptions goes back to at least BarnowCainGoldberger1980. They developed the “linear control function estimator”, which is based on an early parametric version of the now standard unconfoundedness assumption. HeckmanRobb1985, HeckmanHotz1989, and HeckmanVytlacil2007part2 all provide detailed discussions of this estimator. It was also one of the main estimators used in LaLonde1986. StockWatson2011 provide a textbook description of it on pages 230--233 and pages 250--251. AngristPischke2009 also discuss it at the end of their section 3.2.1. Also see Frolich2008. Note that this earlier analysis was based on mean independence assumptions, while the analysis in our paper only uses linear projections. More recently, HuenermundLouw2020 remind researchers that most control variables are likely endogenous and hence their coefficients should not be interpreted as causal.
Although control variables are often thought to be endogenous, the literature on sensitivity analysis generally either assumes the controls are exogenous or uses a method called “residualization.” This includes AltonjiElderTaber2005, AltonjiElderTaber2008, Krauth2016, and Oster2019. Imbens2003 starts from the standard unconfoundedness assumption which allows endogenous controls, but in his parametric assumptions (see his likelihood equation on page 128) he assumes that the unobserved omitted variable is independent of the observed covariates. AET2019 propose an approach to allow for endogenous controls based on imposing a factor model on all covariates, observable and unobservable. Their approach and ours impose different assumptions and therefore are not nested. CinelliHazlett2020 develop a regression sensitivity analysis based on partial R-squareds. Their results do not rely on endogenous controls, but they do not define any measures of selection and do not work with symmetric ratio parameters like AltonjiElderTaber2005, Oster2019, or us, and hence their results do not allow researchers to compare selection on observables with unobservables as in AltonjiElderTaber2005, Oster2019, and our results.
Our companion paper DiegertMastenPoirier2025 uses an axiomatic framework to formally prove that residualization, the literature's most popular approach to allowing for endogenous controls, can lead to incorrect conclusions about robustness. DeLucaMagnusPeracchi2019, CinelliHazlett2020, Basu2022, and MastenPoirier2025 discuss a variety of additional limitations with the existing approaches to assessing sensitivity to omitted variables in linear models. The contribution of the present paper is not to survey or analyze properties of previously proposed methods, but rather to derive new identification results based on a new set of sensitivity parameters. These identification results can then be used to perform a sensitivity analysis to assess the impact of omitted variables on one's regression results. In contrast, our companion paper DiegertMastenPoirier2025 does not derive any identification results or propose any new sensitivity analyses. That paper instead focuses on comparing different definitions of sensitivity parameters, including the ones we propose in the present paper as well as a variety of parameters proposed elsewhere in the literature.
Thus far we have discussed the linear regression based literature. Nonparametric methods, like those based on the standard unconfoundedness assumption (for example, chapter 12 in ImbensRubin2015), often allow for endogenous controls. For this reason, several papers that assess sensitivity to unconfoundedness also allow for endogenous controls. This includes Rosenbaum1995, Rosenbaum2002, Tan2006, and MastenPoirier2018, among others. These methods, however, do not provide formal results for calibrating their sensitivity parameters based on comparing selection on observables with selection on unobservables. These methods also focus on binary or discrete treatments, whereas the analysis in our paper can be used for continuous treatments as well.
For random vectors $A$ and $B$, let $\operatorname*{cov}(A,B)$ be the $\text{dim}(A) \times \text{dim}(B)$ matrix whose $(i,j)$th element is $\operatorname*{cov}(A_i, B_j)$. Define $A^{\perp B} \coloneqq A - \operatorname*{cov}(A,B)\operatorname*{var}(B)^{-1}B$, the intercept plus the residual from a regression of $A$ on $(1,B)$. For a scalar random variable $A$, let $R_{A \sim B \mathrel{\mathsmaller{\bullet}} C}^2$ denote the R-squared from a regression of $A^{\perp C}$ on $(1,B^{\perp C})$; this is called the partial R-squared (of $A$ on $B$, partialling out $C$).
Let $Y$ and $X$ be observed scalar variables. Let $W_1$ be a vector of observed covariates of dimension $d_1$ and $W_2$ be an unobserved vector of dimension $d_2$. Let $W \coloneqq (W_1,W_2)$. Consider the population linear regression of $Y$ on $X$, $W_1$, $W_2$, and a constant. Let $\beta_\text{long}$ denote the coefficient on $X$, while $(\gamma_1,\gamma_2)$ denote the coefficients on $(W_1,W_2)$. The setup here is similar to our companion paper DiegertMastenPoirier2025. We make the following assumption to ensure these linear regression coefficients are well-defined.
We can write
The term $Y^{\perp X,W}$ denotes the linear projection residual plus the intercept. By construction, it is uncorrelated with $(X,W)$. We take as given that $\beta_\text{long}$ is the parameter of interest. This interest may come from an underlying causal structure, where $\beta_\text{long}$ has an interpretation as an average of treatment effects. For example, section (ref) below discusses how unconfoundedness, difference-in-differences, or instrumental variables as identification strategies may yield a causal interpretation for $\beta_\text{long}$. These interpretations may or may not require associated potential outcomes to be linear in $X$. Ultimately, the results we present for $\beta_\text{long}$ do not depend on the reason for the interest in this parameter, and we present our analysis without assuming any particular causal structure.
Next consider the “first-stage” linear regression of $X$ on $W_1$, $W_2$, and a constant. Let $(\pi_1,\pi_2)$ denote the coefficients on $(W_1,W_2)$. We can write
The term $X^{\perp W}$ denotes the linear projection residual plus the intercept. By construction it is uncorrelated with each covariate in $(W_1,W_2)$. Although equation (ref) is not necessarily causal, we can think of $\pi_1$ as representing “selection on observables” while $\pi_2$ represents “selection on unobservables.” The following is thus a natural baseline assumption.
Let $\beta_\text{med}$ denote the coefficient on $X$ in the linear regression of $Y$ on $(1,X,W_1)$. With no selection on unobservables, we have the following result.
This result shows why Assumption (ref) can also be interpreted as a “selection on observables” assumption: adjusting linearly for the observable $W_1$ is sufficient to recover $\beta_\text{long}$. Note that this result allows for endogenous controls, in the sense that the observed covariates $W_1$ can be arbitrarily correlated with the unobserved covariates $W_2$. But it restricts the relationship between $(X,W_1,W_2)$ in such a way that we can still point identify $\beta_\text{long}$ even though $W_1$ and $W_2$ are arbitrarily correlated. The coefficient $\gamma_1$ on the observed controls, however, is completely unidentified. Note that Proposition (ref) is unsurprising and is not the main contribution of this paper; we include it for completeness because we have not seen it formally stated in any previous references, especially the second part that $\gamma_1$ is completely unidentified.
In practice, we are often worried that the no selection on unobservables assumption does not hold. For example, consider the regression of wages on education discussed in section (ref). Even with a rich set of observed covariates $W_1$, like family background variables, researchers typically suspect that there may still be omitted variables that affect both wages and education, and which could be correlated with observed variables like family background. We therefore develop a new approach to assess the importance of this no selection on unobservables assumption in the next section.
There are many ways to relax Assumption (ref) ($\pi_2 = 0$) and allow for some selection on unobservables. A key challenge is to construct measures of selection on unobservables and observables that are interpretable and comparable, and therefore allow researchers to reason about their relative magnitudes. In this section we propose a new measure of selection, argue for its interpretability, and then show how to use it to perform a sensitivity analysis to the selection on observables assumption.
We begin by focusing on the relationship between treatment and the covariates, observed and unobserved. Later, in section (ref), we will extend our analysis to also consider the relationship between outcomes and the covariates. Several previous approaches in the literature use a single sensitivity parameter to measure the combined impact of omitted variables on outcomes and treatment; this leads to a variety of issues and limitations, however, as discussed in the papers cited in section (ref), and hence we instead consider these two relationships separately. To that end, recall the regression of treatment on a constant and all the covariates: \[ X = \pi_1' W_1 + \pi_2' W_2 + X^{\perp W}. \tag{\ref{eq:XprojectionW1W2}} \] We measure selection on unobservables by $\sqrt{\operatorname*{var}(\pi_2'W_2)}$, the standard deviation of the index of unobservables in this regression. We measure selection on observables by $\sqrt{\operatorname*{var}(\pi_1'W_1)}$, the standard deviation of the index of observables in this regression. We compare them by taking the ratio
This is a relative measure that measures the impact of unobservables as a multiple of the impact of observables. Consequently, in models where the included covariates $W_1$ are very important for treatment selection even a small value of $r_X$ might be considered to be a large impact of the unobservables; we discuss this further in section (ref).
Both the numerator and denominator of $r_X$ are invariant to invertible linear transformations of $W_1$ or $W_2$, including rescalings, since the indices $\pi_1'W_1$ and $\pi_2'W_2$ are both invariant with respect to invertible linear transformations. This invariance ensures that $r_X$ is a unit-free measure of the relative magnitude of selection on unobservables to selection on observables. $r_X = 0$ corresponds to baseline assumption (ref) ($\pi_2 = 0$) while $r_X > 0$ allows for some selection on unobservables.
This measure has two additional interpretations. First, $r_X$ has an R-squared interpretation. Specifically, for scalar $W_1$ and $W_2$ we have
The numerator is the difference in the R-squared from the regression in (ref) and the short R-squared that only includes $W_1$. The denominator is the same but switching the roles of $W_1$ and $W_2$. For vector covariates,
Second, $r_X$ can have a causal interpretation. Suppose $W_1$ and $W_2$ are scalars. Let $X(w_1,w_2)$ denote potential treatments. Assume this function is linear in $w_1$ and $w_2$: \[ X(w_1,w_2) = \pi_1 w_1 + \pi_2 w_2 + V \] for some structural unobservable $V$. Assume $\operatorname*{cov}(W,V) = \operatorname*{cov}(W, X(w_1,w_2)) = 0$. Then $ \sqrt{\operatorname*{var}(\pi_2 W_2)}$ is the causal effect of a one standard deviation increase in the unobserved variable on $X$. Likewise, $\sqrt{\operatorname*{var}(\pi_1 W_1)}$ is the causal effect of a one standard deviation increase in the observed variable on $X$. And the measure $r_X$ is the magnitude of the ratio of these two causal effects.
Having defined and interpreted our measure of selection on unobservables, we next move on to study identification. In section (ref) we state our main identification results based on $r_X$ only. In section (ref) we extend these results to allow researchers to also restrict the impact of omitted variables on the outcome variable. In section (ref) we make several remarks regarding interpretation and calibration of the sensitivity parameters. In section (ref), we extend these results to allow researchers to also restrict the magnitude of control endogeneity. Finally, in section (ref) we briefly discuss estimation and inference.
Recall from section (ref) that our parameter of interest is $\beta_\text{long}$, the OLS coefficient on $X$ in the long regression of $Y$ on $(1,X,W_1,W_2)$. Since $W_2$ is not observed, we cannot compute this regression from the data. However, we can compute $\beta_\text{med}$, the OLS coefficient on $X$ in the medium regression of $(1,X,W_1)$. The difference between these two regression coefficients is given by the classic omitted variable bias formula. To bound this difference, we restrict the selection ratio $r_X$ described in section (ref). Specifically, consider the following assumption.
Following our discussion in section (ref), this assumption is invariant to any invertible linear transformations of $W_1$ or $W_2$, and thus $\bar{r}_X$ is a unit-free sensitivity parameter. Assumption (ref) can also be written as \[ \sqrt{ \operatorname*{var}(\pi_2' W_2) } \leq \bar{r}_X \cdot \sqrt{ \operatorname*{var}(\pi_1' W_1) }, \] which means that the association between treatment $X$ and a one standard deviation increase in the index of unobservables is at most $\bar{r}_X$ times the association between treatment and a one standard deviation increase in the index of observables. The baseline model of section (ref) corresponds to the case $\bar{r}_X = 0$, since it implies $\pi_2 = 0$. We relax the baseline model by considering values $\bar{r}_X > 0$.
Let $\mathcal{B}_I(\bar{r}_X)$ denote the identified set for $\beta_\text{long}$ under the positive definite variance assumption (ref) and the restriction (ref). In particular, this identified set does not impose any restrictions on the impact of omitted variables on the outcome variable. Let \[ \underline{B}(\bar{r}_X) \coloneqq \inf \mathcal{B}_I(\bar{r}_X) \qquad \text{and} \qquad \overline{B}(\bar{r}_X) \coloneqq \sup \mathcal{B}_I(\bar{r}_X) \] denote its greatest lower bound and least upper bound. Our first main result, Theorem (ref) below, provides simple, closed form expressions for these sharp bounds. For this theorem, we also use the following assumption.
This assumption is not necessary, but it simplifies the proofs. Moreover, this assumption usually holds in practice since it holds whenever $\beta_\text{short} \neq \beta_{\text{med}}$, where $\beta_\text{short}$ is the coefficient on $X$ in the short OLS regression of $Y$ on $(1,X)$.
Appendix (ref) gives a sketch derivation of and intuition for this result, while the complete formal derivation and sharpness proof is in Appendix (ref). Theorem (ref) characterizes the largest and smallest possible values of $\beta_\text{long}$ when some selection on unobservables is allowed, the observed covariates are allowed to be arbitrarily correlated with the unobserved covariate, and we make no restrictions on the coefficients in the outcome equation. In fact, we prove that, with the exception of at most three singletons, the interval $[\underline{B}(\bar{r}_X), \overline{B}(\bar{r}_X)]$ is the identified set for $\beta_\text{long}$ under these assumptions. Here we focus on the smallest and largest elements to avoid technical digressions that are unimportant for applications.
There are two important features of Theorem (ref): First, it only requires researchers to reason about one sensitivity parameter, unlike some existing approaches (our generalizations in sections (ref) and (ref) use additional sensitivity parameters to allow researchers to impose further restrictions). Second, it allows for arbitrarily endogenous controls since it imposes no restrictions on the association between $W_1$ and $W_2$. So this result allows researchers to examine the impact of selection on unobservables on their baseline results without also having to reason about the magnitude of endogenous controls.
Since Theorem (ref) provides explicit expressions for the bounds, we can immediately derive a few of their properties. First, when $\bar{r}_X = 0$, the bounds collapse to $\beta_\text{med}$, the point estimand from the baseline model with no selection on unobservables. Thus, we recover the baseline model as a special case. For small values of $\bar{r}_X > 0$, the bounds are no longer a singleton, but their width increases continuously as $\bar{r}_X$ increases away from zero. The rate at which the bounds increase depends on just a few features of the data: All can be derived from the variance matrix of $(Y,X,W_1)$. We also see that the bounds are symmetric around $\beta_\text{med}$. Finally, the bounds are finite only if $\bar{r}_X < \sqrt{1 - R^2_{X \sim W_1}}$.
In practice, researchers often ask:
We can use Theorem (ref) to answer this question. Suppose in the baseline model we find $\beta_\text{med} \geq 0$. We are concerned, however, that $\beta_\text{long} \leq 0$, in which case our positive finding is driven solely by selection on unobservables. Define \[ \bar{r}_X^\text{bp} \coloneqq \sup \{ \bar{r}_X \geq 0 : b \geq 0 \text{ for all } b \in \mathcal{B}_I(\bar{r}_X) \}. \] This value is called a breakdown point. It is the largest amount of selection on unobservables we can allow for while still concluding that $\beta_\text{long}$ is nonnegative. The breakdown point when $\beta_\text{med} \leq 0$ can be defined analogously.
The breakdown point described in Corollary (ref) characterizes the magnitude of selection on unobservables relative to selection on observables needed to overturn one's baseline findings. Corollary (ref) explicitly shows that this breakdown point depends on just two features of the observed data: The relationship between treatment and the outcome, after adjusting for the observed covariates, and the first stage relationship between treatment and the observed covariates. In particular, as the covariate adjusted relationship between outcomes and treatment strengthens, the breakdown point increases too. In contrast, as the relationship between treatment and covariates strengthens, the breakdown point decreases. This follows since we are using effects of the observed covariates to calibrate the magnitude of selection on unobservables. So when these observed covariates are strongly related to treatment, we need relatively less selection on unobservables to overturn our baseline findings. We further discuss this point in section (ref).
First, we recommend that researchers present estimates of the breakdown point $\bar{r}_X^\text{bp}$ as a scalar measure of the robustness of their results. This can be displayed alongside the main result point estimates, as in Table (ref) in our empirical application. Second, we recommend that researchers present estimates of the bounds described in Theorem (ref) as a function of $\bar{r}_X$. Researchers may also find it useful to present estimates of the calibration parameters $\rho_k$ that we describe in section (ref). We illustrate all of these recommendations in our empirical application in section (ref).
In some empirical settings researchers may be willing to restrict the impact of unobservables on outcomes. To do this, by analogy to $r_X$, define \[ r_Y \coloneqq \frac{\sqrt{\operatorname*{var}(\gamma_2'W_2)}}{\sqrt{\operatorname*{var}(\gamma_1'W_1)}}. \] This is the ratio of the standard deviation of the index of unobservables in the outcome equation (ref) to the standard deviation of the index of observables in the same equation. Thus $r_Y$ measures the relative impact of unobservables and observables on outcomes, whereas $r_X$ measures their relative impact on treatment. Note that, like the discussion of $r_X$ in section (ref), $r_Y$ can also be interpreted as a ratio of causal effects under a linearity assumption on potential outcomes. We now consider a bound on this ratio $r_Y$.
Assumption (ref) has a similar interpretation as (ref): It says that the association between the outcome and a one standard deviation increase in the index of unobservables is at most $\bar{r}_Y$ times the association between the outcome and a one standard deviation increase in the index of observables. Like $r_X$, the parameter $r_Y$ is invariant to invertible linear transformations of $W_1$ and $W_2$, making $\bar{r}_Y$ a unit-free sensitivity parameter.
In our next result, which uses restrictions on both $r_X$ and $r_Y$, we assume that the omitted variable of interest $W_2$ is a scalar, and then normalize its variance to 1. We leave the extension to vector $W_2$ to future work. Let $\mathcal{B}_I(\bar{r}_X,\bar{r}_Y)$ denote the identified set for $\beta_\text{long}$ under this scalar $W_2$ assumption, (ref), (ref), and (ref). Unlike the identified set of Theorem (ref), this set is less analytically tractable. We characterize it in Appendix (ref). In this section we show that it is nonetheless very easy to numerically implement a breakdown analysis, which is often researchers' main focus.
Suppose we are interested in the robustness of the conclusion that $\beta_\text{long} \geq \underline{b}$ for some known scalar $\underline{b}$. For example, $\underline{b} = 0$. Define the function \[ \bar{r}_Y^\text{bf}(\bar{r}_X, \underline{b}) \coloneqq \sup \{ \bar{r}_Y \geq 0 : b \geq \underline{b} \text{ for all } b \in \mathcal{B}_I(\bar{r}_X, \bar{r}_Y) \}. \] This function is a breakdown frontier (MastenPoirier2019BF), which captures trade-offs in the relative impacts of $\bar{r}_X$ and $\bar{r}_Y$ on the identified set. In particular, we can use it to define the set \[ \text{RR} \coloneqq \{ (\bar{r}_X, \bar{r}_Y) \in \ensuremath{\mathbb{R}}_{\geq 0}^2 : \bar{r}_Y \leq \bar{r}_Y^\text{bf}(\bar{r}_X,\underline{b}) \}. \] This set is called the robust region because the conclusion of interest, $\beta_\text{long} \geq \underline{b}$, holds for any combination of sensitivity parameters in this region. The size of this region is therefore a measure of the robustness of our baseline conclusion.
Although we do not have a closed form expression for the smallest and largest elements of $\mathcal{B}_I(\bar{r}_X, \bar{r}_Y)$, our next main result shows that we can still easily compute the breakdown frontier numerically. To state the result, we first define some additional notation. Let $\| \cdot \|$ denote the Euclidean norm. Define the set
and the functions
For simplicity, we also normalize the treatment variable so that $\operatorname*{var}(X) = 1$ and the covariates so that $\operatorname*{var}(W_1) = I$. All of the results below can be rewritten without these normalizations at the cost of additional notation. We can now state our second main result.
Theorem (ref) shows that the breakdown frontier can be computed as the solution to a smooth optimization problem. Importantly, this problem only requires searching over a 4-dimensional space. In particular, this dimension does not depend on the dimension of the covariates $W_1$. Consequently, it remains computationally feasible even with a large number of observed covariates, as is often the case in empirical practice. For example, the results for our empirical application take about 15 seconds to compute on a standard machine.
Theorem (ref) makes several minor technical assumptions. In particular, it assumes $\operatorname*{cov}(W_1, Y)$ and $\operatorname*{cov}(W_1, X)$ are linearly independent for simplicity. Moreover, the theorem assumes $d_1 \geq 2$, requiring at least two observed covariates in $W_1$. This is not restrictive since the purpose of this result is primarily to show that the optimization problem does not depend on the dimension of $W_1$. If $d_1 = 1$ then the breakdown frontier can instead be easily computed using equation (ref) in the appendix, which only requires searching over a 3-dimensional space.
To assess the impact of different choices of $(\bar{r}_X, \bar{r}_Y)$ on the robustness of their results, researchers can plot sample analog estimates of the function $\bar{r}_Y^\text{bf}(\cdot, \underline{b})$ (for example, see the solid line in the right plots of figures (ref) and (ref) in section (ref) below). The conclusion of interest $\beta_\text{long} \geq \underline{b}$ holds for all pairs $(\bar{r}_X, \bar{r}_Y)$ below this function. In practice it is also useful to report scalar measures of robustness to omitted variables. One way to do this while also incorporating restrictions on the impact of the omitted variables on outcomes is to set $\bar{r}_X = \bar{r}_Y$. This assumption says that the maximal impact of unobservables relative to observables is the same for outcomes as it is for treatment. Under this common maximal impact assumption, the breakdown point for the common value of the sensitivity parameter is \[ \bar{r}^\text{bp}(\underline{b}) \coloneqq \sup\{r \geq 0: b \geq \underline{b} \text{ for all } b \in \mathcal{B}_I(r,r)\}. \] Let $\bar{r}^\text{bp} = \bar{r}^\text{bp}(0)$. Our second main recommendation is that researchers report estimates of this breakdown point for some value $\underline{b}$ of interest. $\bar{r}^\text{bp}$ can be viewed as a less conservative version of $\bar{r}_X^\text{bp}$, the breakdown point that only imposes restrictions on how treatment is related to the omitted variables. In particular, note that $\bar{r}_X^\text{bp} \leq \bar{r}^\text{bp}$ always holds. The common maximal impact breakdown point $\bar{r}^\text{bp}(\underline{b})$ can be directly computed from the function $\bar{r}^\text{bf}_Y(\bar{r}_X, \underline{b})$ by solving $\bar{r}^\text{bf}_Y(\bar{r}_X, \underline{b}) = \bar{r}_X$ numerically.
Finally, note that sharp bounds on the identified set $\mathcal{B}_I(\bar{r}_X,\bar{r}_Y)$ can be easily computed by using a similar duality relationship between the breakdown frontier and the identified set as described in Proposition 1 of MastenPoirier2023.
Having stated our main identification results, we next make several remarks regarding how to interpret the magnitudes of these parameters.
The sensitivity parameters $\bar{r}_X$ and $\bar{r}_Y$ are relative sensitivity parameters, in the sense that they compare the importance of the unobservables $W_2$ relative to the importance of the observed covariates $W_1$. Thus far we have assumed that $W_1$ includes all observed covariates. Here we extend the analysis to allow researchers to select which covariates to use in the sensitivity parameters. Hence the interpretation of the sensitivity parameters depends on the choice of covariates that we calibrate against. Put differently, when we say that we compare “selection on unobservables to selection on observables,” which observables do we mean?
To answer this, we split the observed covariates into two groups: (1) The control covariates, which we label $W_0$, and (2) The calibration covariates, which we continue to label $W_1$. Write equation (ref) as \[ Y = \beta_\text{long} X + \gamma_0' W_0 + \gamma_1' W_1 + \gamma_2' W_2 + Y^{\perp X,W} \tag{\ref{eq:outcome}$^\prime$} \] where $W \coloneqq (W_0,W_1,W_2)$. Likewise, write equation (ref) as \[ X = \pi_0' W_0 + \pi_1' W_1 + \pi_2' W_2 + X^{\perp W}. \tag{\ref{eq:XprojectionW1W2}$^\prime$} \] The key difference from our earlier analysis is that, like in Assumption (ref), we will continue to only compare $\pi_1$ with $\pi_2$. That is, we only compare the omitted variable to the observed variables in $W_1$; we do not use $W_0$ for calibration. A similar remark applies to (ref).
This distinction between control and calibration covariates is useful because in many applications we do not necessarily think the omitted variables have similar explanatory power as all of the observed covariates included in the model. For example, in our empirical application in section (ref), we include state fixed effects as control covariates, but we do not use them for calibration.
We next briefly describe how to generalize our identification results to account for this distinction. By the FWL theorem, a linear projection of $Y^{\perp W_0}$ onto $(1,X^{\perp W_0}, W_1^{\perp W_0}, W_2^{\perp W_0})$ has the same coefficients as equation (ref). Likewise for a linear projection of $X^{\perp W_0}$ onto $(1,W_1^{\perp W_0}, W_2^{\perp W_0})$. Hence we can write
Therefore our earlier results continue to hold when $(Y,X,W_1,W_2)$ are replaced with ($Y^{\perp W_0}$, $X^{\perp W_0}$, $W_1^{\perp W_0}$, $W_2^{\perp W_0}$).
Here we sketch a data-driven approach to calibrating the magnitude of the sensitivity parameter $\bar{r}_X$, and therefore for assessing the robustness of one's results to omitted variables. By analogy to $r_X = \sqrt{ \operatorname*{var}(\pi_2'W_2)} / \sqrt{ \operatorname*{var}(\pi_1'W_1)}$, define \[ \rho_k \coloneqq \frac{ \sqrt{\operatorname*{var}(\pi_{1,\text{med},k} W_{1k})} }{ \sqrt{ \operatorname*{var}(\pi_{1,\text{med},-k}'W_{1,-k} ) } } \] for $k=1,\ldots,d_1$, where $\pi_{1,\text{med}} = (\pi_{1,\text{med},k}, \pi_{1,\text{med},-k})$ is the coefficient on the observed covariates $W_1 = (W_{1k}, W_{1,-k})$ from OLS of treatment $X$ on $(1,W_1)$. $\rho_k$ measures the importance of variable $k$ relative to all other observed covariates. Moreover, unlike $r_X$, the $\rho_k$ values are point identified from the data. We suggest that researchers compare their estimates of the breakdown point $\bar{r}_X^\text{bp}$ against estimates of the values $\rho_1,\ldots,\rho_{d_1}$. We illustrate this approach in our empirical analysis in section (ref). Similar approaches to empirically calibrating sensitivity parameters were previously proposed in HosmanHansenHolland2010, HsuSmall2013, MastenPoirier2018, and CinelliHazlett2020. Importantly, as HosmanHansenHolland2010 discuss in detail, values like $\rho_k$ are “reference points for speculation” about the true, but unknown and completely unidentified value of $r_X$. Put differently: There is no logical guarantee that $r_X$ will be smaller than any of the $\rho_k$'s. Instead, their purpose is simply to help us use data to reason about plausible values of the true, but unknown sensitivity parameter $r_X$. Finally, here we defined $\rho_k$ by comparing a single covariate against all others. More generally one could compare the impact of a group of covariates against all others; we omit this extension for brevity.
Our results thus far have allowed for arbitrarily endogenous controls. In this section we present more general identification results that allow researchers to impose restrictions on the correlation between the observed covariates and the omitted variable. There are at least two reasons why researchers may be interested in imposing such restrictions: (1) First, it allows us to compare bounds obtained under exogenous controls with those obtained under arbitrarily endogenous controls. This allows us to assess the impact of endogenous controls on the robustness of one's results. (2) Second, when the observed controls are believed to be sufficiently endogenous, it may be plausible to impose a lower bound on their correlation.
To motivate our formal results, note that for scalar $W_2$ with $\operatorname*{var}(W_2) = 1$ we can write the omitted variable bias as a function of the coefficient $\gamma_2$ on $W_2$ in the long regression outcome equation (ref) and the coefficient $\pi_2$ on $W_2$ in the selection equation (ref) as follows:
where $R_{W_2 \sim W_1}^2$ denotes the population $R^2$ in a linear regression of the unobserved $W_2$ on the observed covariates $(1,W_1)$. Our restrictions using $\bar{r}_X$ and $\bar{r}_Y$ imply bounds on $\pi_2$ and $\gamma_2$. The remaining parameter in equation (ref) is $R_{W_2 \sim W_1}^2$. We consider the following restrictions directly on this R-squared.
Our results in section (ref) can be thought of as the agnostic case $[\underline{c}, \bar{c}] = [0,1]$. We first consider restrictions under Assumption (ref) and a bound on $r_X$. We then consider restrictions on $r_Y$ as well. We conclude this section by discussing an approach to assessing control endogeneity and to calibrating the values $\underline{c}$ and $\bar{c}$.
Remark on extension to non-calibration covariates. In section (ref) we discussed how to include covariates $W_0$ that are not used for calibration. When including these variables, the $\underline{c}$ and $\bar{c}$ variables should be interpreted as bounds on $R_{W_2 \sim W_1 \mathrel{\mathsmaller{\bullet}} W_0}$, the square root of the R-squared from the regression of $W_2$ on $W_1$ after partialling out $W_0$.
Let $\mathcal{B}_I(\bar{r}_X,\underline{c},\bar{c})$ denote the identified set for $\beta_\text{long}$ under the positive definite variance assumption (ref), the restriction (ref) on $r_X$, Assumption (ref) on $R_{W_2 \sim W_1}^2$, and when $W_2$ is a scalar with $\operatorname*{var}(W_2)$ normalized to 1. Let \[ \underline{B}(\bar{r}_X, \bar{c}) \coloneqq \inf \mathcal{B}_I(\bar{r}_X,\underline{c}, \bar{c}) \qquad \text{and} \qquad \overline{B}(\bar{r}_X, \bar{c}) \coloneqq \sup \mathcal{B}_I(\bar{r}_X,\underline{c}, \bar{c}) \] denote its greatest lower bound and least upper bound. Like Theorem (ref), Theorem (ref) below provides simple, closed form expressions for these sharp bounds.
Appendix equation (ref) defines the function $\bar{z}_X(\bar{r}_X,\underline{c},\bar{c})$; its exact form is unimportant here. The sensitivity parameters will only affect the bounds via this function.
Theorem (ref) can be obtained as a special case of Theorem (ref) by setting $\underline{c} = 0$ and $\overline{c} = 1$, and hence the two theorems have similar interpretations: Theorem (ref) characterizes the largest and smallest possible values of $\beta_\text{long}$ when some selection on unobservables is allowed and the controls are allowed to be partially but not arbitrarily endogenous. We also make no restrictions on the coefficients in the outcome equation. As before, with the exception of at most three singletons, the interval $[\underline{B}(\bar{r}_X,\underline{c},\bar{c}), \overline{B}(\bar{r}_X,\underline{c}, \bar{c})]$ is the identified set for $\beta_\text{long}$ under these assumptions.
Earlier we saw that $\bar{r}_X < 1$ is necessary for the bounds of Theorem (ref) to be finite. Theorem (ref) shows that, if we are willing to make assumptions on $R^2_{W_2 \sim W_1}$ through $[\underline{c},\bar{c}]$, then we can allow for $\bar{r}_X > 1$ while still obtaining finite bounds since $\bar{z}_X(\bar{r}_X,\underline{c},\bar{c})$ is finite whenever $\bar{r}_X < \bar{c}^{-1}$. Thus there is a trade-off between (i) the magnitude of selection on unobservables we can allow for and (ii) the magnitude of control endogeneity. We can use breakdown frontiers to summarize these trade-offs. Specifically, when $\beta_\text{med} \geq 0$, define \[ \bar{r}_X^\text{bf}(\underline{c},\bar{c}) \coloneqq \sup \{ \bar{r}_X \geq 0 : b \geq 0 \text{ for all } b \in \mathcal{B}_I(\bar{r}_X, \underline{c}, \bar{c}) \}. \] Then the conclusion that $\beta_\text{long} \geq 0$ is robust to any combination $(\bar{r}_X, \underline{c}, \bar{c})$ such that $\bar{r}_X \leq \bar{r}_X^\text{bf}(\underline{c},\bar{c})$. Put differently, for any fixed $\underline{c}$ and $\bar{c}$, $\bar{r}_X^\text{bf}(\underline{c},\bar{c})$ is a breakdown point: It is the largest magnitude of selection on unobservables relative to selection on observables that we can allow for while still concluding that our parameter of interest is nonnegative.
Next we generalize Theorem (ref). Let \[ \bar{r}_Y^\text{bf}(\bar{r}_X,\underline{c},\bar{c}, \underline{b}) \coloneqq \sup \{ \bar{r}_Y \geq 0 : b \geq \underline{b} \text{ for all } b \in \mathcal{B}_I(\bar{r}_X, \bar{r}_Y, \underline{c},\bar{c}) \}. \]
Theorem (ref) is the special case obtained by setting $[\underline{c},\bar{c}] = [0,1]$.
Assumption (ref) is a constraint on the covariance between the observed calibration covariates and the unobserved covariate. Specifically, when including non-calibration variables, it is a bound on $R_{W_2 \sim W_1 \mathrel{\mathsmaller{\bullet}} W_0}$. So what values of this parameter should be considered large, and what values should be considered small? One way to calibrate this parameter is to compute \[ c_k \coloneqq R_{W_{1k} \sim W_{1,-k} \mathrel{\mathsmaller{\bullet}} W_0} \] for each covariate $k$ in $W_1$. That is, compute the square root of the population R-squared from the regression of $W_{1k}$ on the rest of the calibration covariates $W_{1,-k}$, after partialling out the control covariates $W_0$. These numbers tell us two things. First, if many of these values are nonzero and large, we may worry that the exogenous controls assumption fails. That is, if $W_2$ is in some way similar to the observed covariates $W_1$, then we might expect that $R_{W_2 \sim W_1 \mathrel{\mathsmaller{\bullet}} W_0}$ is similar to some of the $c_k$'s. So this gives us one method for assessing the plausibility of exogenous controls. Second, we can use the magnitudes of these values to calibrate our choice of $\bar{c}$, in analysis based on Theorem (ref) or Theorem (ref). For example, you could choose $\underline{c} = \min_k c_k$ and $\bar{c} = \max_k c_k$. Table (ref) in our empirical analysis shows sample analog estimates of $c_k^2$.
Thus far we have described population level identification results. In practice, we only observe finite sample data. Our identification results depend on the observables $(Y,X,W_1)$ solely through their covariance matrix. In our empirical analysis in section (ref), we apply our identification results by using sample analog estimators that replace $\operatorname*{var}(Y,X,W_1)$ with a consistent estimator $\widehat{\operatorname*{var}}(Y,X,W_1)$. For example, we let $\widehat{\beta}_\text{med}$ denote the OLS estimator of $\beta_\text{med}$, the coefficient on $X$ in the medium regression of $Y$ on $(1,X,W_1)$. We expect the corresponding asymptotic theory for estimation and inference on the bound functions to be straightforward, but for brevity we do not develop it in this paper.
While the results in section (ref) are stated under general conditions, empirical researchers are often interested in assessing sensitivity to omitted variables in causal models. In this section, we review how $\beta_\text{long}$ in equation (ref) can be interpreted in the context of four specific causal models. These models are based on three identification strategies: Unconfoundedness, difference-in-differences, and instrumental variables. Here we focus on simple models, but our analysis can be used anytime the causal parameter of interest can be written as the coefficient on a treatment variable in a long regression of the form in equation (ref).
Recall that $Y$ denotes the realized outcome, $X$ denotes treatment, $W_1$ denotes the observed covariates, and $W_2$ denotes the unobserved variables of concern. Let $Y(x)$ denote potential outcomes, where $x$ is any logically possible value of treatment. Assume this potential outcome has the following form:
where $(\beta_c,\gamma_1,\gamma_2)$ are unknown constants. The parameter of interest is $\beta_c$, the causal effect of treatment on the outcome. $U$ is an unobserved random variable. Suppose the realized outcome satisfies $Y = Y(X)$. Consider the following assumption.
This assumption says that, after partialling out the observed covariates $W_1$ and the unobserved variables $W_2$, treatment is uncorrelated with the unobserved variable $U$. This model has two unobservables, which are treated differently via this assumption. We call $W_2$ the confounders and $U$ the non-confounders. $W_2$ are the unobserved variables which, when omitted, may cause bias. In contrast, as long as we adjust for $(W_1,W_2)$, omitting $U$ does not cause bias. Note that, given equation (ref), linear latent unconfoundedness can be equivalently written as $\text{corr}(X^{\perp W_1,W_2}, Y(x)^{\perp W_1,W_2}) = 0$ for all logically possible values of treatment $x$.
Linear latent unconfoundedness can be interpreted as a linear parametric version of the nonparametric latent unconfoundedness assumption
for all logically possible values of $x$.
The following result shows that, in this model, the causal effect of $X$ on $Y$ can be obtained from $\beta_\text{long}$, the coefficient on $X$ in the long regression described in equation (ref).
Since $W_2$ is unobserved, however, this result cannot be used to identify $\beta_c$. Instead, suppose we believe the no selection on unobservables assumption (ref). Recall that this assumption says that $\pi_2 = 0$, where $\pi_2$ is the coefficient on $W_2$ in the OLS estimand of $X$ on $(1,W_1,W_2)$. Under this assumption, we obtain the following result. Recall that $\beta_\text{med}$ denotes the coefficient on $X$ in the medium regression of $Y$ on $(1,X,W_1)$.
The selection on observables assumption (ref) is usually thought to be quite strong, however. Nonetheless, since $\beta_c = \beta_\text{long}$, our results in section (ref) can be used to assess sensitivity to selection on unobservables.
In the previous section we considered a linear parametric version of the unconfoundedness identification strategy. Here we consider a nonparametric version, which allows us to extend our analysis to models with heterogeneous treatment effects. Let $Y(x)$ denote potential outcomes, $X$ be a binary treatment, and $W \coloneqq (W_1,W_2)$ a vector of covariates. Suppose the following two assumptions hold:
For any $w \in \operatorname*{supp}(W)$, let \[ \text{CATE}(w) \coloneqq \ensuremath{\mathbb{E}}[Y(1) - Y(0) \mid W=w] \] denote the conditional average treatment effect. Let $\beta_\text{long}$ be the coefficient in OLS of $Y$ on $(1,X,W_1,W_2)$. Assume $\ensuremath{\mathbb{E}}(X \mid W=w)$ is linear in $w$. That is, we can write \[ \ensuremath{\mathbb{E}}(X \mid W=w) = \pi_1' w_1 + \pi_2' w_2. \] Define \[ \beta_c = \ensuremath{\mathbb{E}}[ \omega(W) \text{CATE}(W)] \] where \[ \omega(w) = \frac{\operatorname*{var}(X \mid W=w)}{\ensuremath{\mathbb{E}}[\operatorname*{var}(X \mid W)]}. \] Under the assumptions above, it follows from Angrist1998 that $\beta_\text{long} = \beta_c$. Thus $\beta_\text{long}$ has a causal interpretation as a weighted average of CATE's.
Now suppose $W_2$ is not observed. Recall that $\beta_\text{med}$ denotes the coefficient on $X$ in the medium regression of $Y$ on $(1,X,W_1)$. If $\pi_2 = 0$, then $\beta_\text{med} = \beta_\text{long}$. But if $\pi_2 \neq 0$, generally $\beta_\text{med} \neq \beta_\text{long}$. We can then use our results in section (ref) to do sensitivity analysis for $\beta_c$.
Let $Y_t(x_t)$ denote potential outcomes at time $t$, where $x_t$ is a logically possible treatment value. For simplicity we do not consider models with dynamic effects. Also suppose $W_{2t}$ is a scalar for simplicity. Suppose
where $V_t$ is an unobserved random variable and $(\beta_c,\gamma_1,\gamma_2)$ are unknown parameters that are constant across units. The classical two way fixed effects model is a special case where
where $A$ is an unobserved random variable that is constant over time, $\delta_t$ is an unobserved constant, and $U_t$ is an unobserved random variable.
Suppose there are two time periods, $t \in \{1,2\}$. Let $Y_t = Y_t(X_t)$ denote the observed outcome at time $t$. For any time varying random variable like $Y_t$, let $\Delta Y = Y_2 - Y_1$. Then taking first differences of the observed outcomes yields \[ \Delta Y = \beta_c \Delta X + \gamma_1' \Delta W_1 + \gamma_2 \Delta W_2 + \Delta V. \] Let $\beta_\text{long}$ denote the OLS coefficient on $\Delta X$ from the long regression of $\Delta Y$ on $(1,\Delta X, \Delta W_1, \Delta W_2)$.
The exogeneity assumption in Proposition (ref) says that $\Delta V$ is uncorrelated with all components of $(\Delta X, \Delta W_1, \Delta W_2)$. A sufficient condition for this is the two way fixed effects assumption (ref) combined with the assumption that the $U_t$ are uncorrelated with $(X_s,W_{1s}, W_{2s})$ for all $t$ and $s$. Given this exogeneity assumption, the only possible identification problem is that $\Delta W_2$ is unobserved. Hence we cannot adjust for this trend variable. If we assume, however, that treatment trends $\Delta X$ are not related to the unobserved trend $\Delta W_2$, then we can point identify $\beta_c$. Specifically, consider the linear projection of $\Delta X$ onto $(1,\Delta W_1, \Delta W_2)$: \[ \Delta X = \pi_1' (\Delta W_1) + \pi_2 (\Delta W_2) + (\Delta X)^{\perp \Delta W_1, \Delta W_2}. \] Using this equation to define $\pi_2$, we now have the following result. Here we let $\beta_\text{med}$ denote the coefficient on $\Delta X$ in the medium regression of $\Delta Y$ on $(1, \Delta X, \Delta W_1)$.
This result implies that $\beta_c$ is point identified when $\pi_2 = 0$. This assumption is a version of common trends, because it says that the unobserved trend $\Delta W_2$ is not related to the trend in treatments, $\Delta X$. Our results in section (ref) allow us to analyze the impacts of failure of this common trends assumption on conclusions about the causal effect of $X$ on $Y$, $\beta_c$. In particular, our results allow researchers to assess the failure of common trends by comparing the impact of observed time varying covariates with the impact of unobserved time varying confounders. In this context, allowing for endogenous controls means allowing for the trend in observed covariates to correlate with the trend in the unobserved covariates. Finally, note that this approach allows researchers to assess sensitivity to common trends even when it is not possible to examine pre-trends; that is, even when there are not multiple time periods where all units are untreated.
Let $Z$ be an observed variable that we want to use as an instrument. Let $Y(z)$ denote potential outcomes, where $z$ is any logical value of the instrument. Assume \[ Y(z) = \beta_c z + \gamma_1' W_1 + \gamma_2' W_2 + U \] where $U$ is an unobserved scalar random variable and $(\beta_c, \gamma_1, \gamma_2)$ are unknown constants. Thus $\beta_c$ is the causal effect of $Z$ on $Y$. In an instrumental variables analysis, this is typically called the reduced form causal effect, and $Y(z)$ are reduced form potential outcomes. Suppose $\operatorname*{cov}(Z^{\perp W_1, W_2}, Y(z)^{\perp W_1, W_2}) = 0$, a linear parametric version of conditional independence between the instrument $Z$ and the unobserved $U$ given $(W_1,W_2)$. Then $\beta_c$ equals the OLS coefficient on $Z$ from the long regression of $Y$ on $(1,Z,W_1,W_2)$. In this model, Proposition (ref) implies that $\beta_c$ is also obtained as the coefficient on $Z$ in the medium regression of $Y$ on $(1,Z,W_1)$, and thus is point identified. In this case, Assumption (ref) is an instrument exogeneity assumption. Our results in section (ref) thus allow us to analyze the impacts of instrument exogeneity failure on conclusions about the reduced form causal effect of $Z$ on $Y$, $\beta_c$.
In a typical instrumental variable analysis, the reduced form causal effect of the instrument on outcomes is not the main effect of interest. Instead, we usually care about the causal effect of a treatment variable on outcomes. The reduced form is often just an intermediate tool for learning about that causal effect. Our analysis in this paper can be used to assess the sensitivity of conclusions about this causal effect to failures of instrument exclusion or exogeneity too. This analysis is somewhat more complicated, however, and so we leave it for a separate paper. Nonetheless, empirical researchers do sometimes examine the reduced form directly to study the impact of instrument exogeneity failure. For example, see Section D7 and Table D15 of Tabellini2020.
Where does culture come from? BFG2020 study the origins of people's preferences for or against government redistribution, intervention, and regulation. They provide the first systematic empirical analysis of a famous conjecture that living on the American frontier cultivated individualism and antipathy to government intervention. The idea is that life on the frontier was hard and dangerous, had little to no infrastructure, and required independence and self-reliance to survive. These features then create cultural change, in particular, leading to “more pervasive individualism and opposition to redistribution”. Overall, BFG2020 find evidence supporting this frontier life conjecture.
BFG2020 provide a wide variety of analyses and contributions. In our analysis, we focus on a subset of their main results, concerning the long run effects of the frontier experience on modern culture. These results are based on an unconfoundedness identification strategy and use linear models. They note that “the main threat to causal identification of $\beta$ lies in omitted variables” and hence they strongly rely on Oster's (2019) sensitivity analysis to “show that unobservables are unlikely to drive our results” (page 2344). As we have discussed in the introduction, however, this method may lead to incorrect conclusions about robustness. In this section, we apply our methods to examine the impact of allowing for endogenous controls on Bazzi et al.'s empirical conclusions about the long run effect of the frontier experience on culture. Overall, while they found that all of their analyses were robust to the presence of omitted variables, we find that their analysis using questionnaire based outcomes are sometimes substantially sensitive to omitted variables, while their analysis using property tax levels and voting patterns is generally robust. We also find suggestive evidence that the controls are endogenous, which highlights the value of sensitivity analysis methods that allow for endogenous controls. We discuss all of these findings in more detail below.
We first describe the variables and data sources. The main units of analysis are counties in the U.S., although we will also use some individual level data. The treatment $X$ is the “total frontier experience” (TFE). This is defined as the number of decades between 1790 and 1890 a county spent “on the frontier”. A county is “on the frontier” if it had a population density less than 6 people per square mile and was within 100 km of the “frontier line”. The frontier line is a line that divides sparse counties (less than or equal to 2 people per square mile) from less sparse counties. By definition, the frontier line changed over time in response to population patterns, but it did so unevenly, resulting in some counties being “on the frontier” for longer than others. Figure 3 in BFG2020 shows the spatial distribution of treatment.
The outcome variable $Y$ is a measure of modern culture. They consider 8 different outcome variables. Since data is not publicly available for all of them, we only look at 5 of these. They can be classified into two groups. The first are questionnaire based outcomes:
These surveys also collected data on individual demographics, specifically age, gender, and race. The second group of outcome variables are based on behavior rather than questionnaire responses:
Next we describe the observed covariates. We partition these covariates into $W_1$ and $W_0$ by following the implementation of Oster's Oster2019 approach in BFG2020. $W_1$, the calibration covariates which are used to calibrate selection on unobservables, is a set of geographic and climate controls: Centroid Latitude, Centroid Longitude, Land area, Average rainfall, Average temperature, Elevation, Average potential agricultural yield, and Distance from the centroid to rivers, lakes, and the coast. $W_0$, the control covariates which are not used to calibrate selection on unobservables, includes state fixed effects. The questionnaire based outcomes use individual level data. For those analyses, we also include age, age-squared, gender, race, and survey wave fixed effects in $W_0$. In BFG2020, they were included in $W_1$. We instead include them in $W_0$ to keep the set of calibration covariates $W_1$ constant across the five main specifications. This allows us to directly compare the robustness of our baseline results across different specifications. We discuss the results that include state fixed effects in $W_1$ in Appendix (ref).
BFG2020 perform a variety of analyses to examine the long run effect of the frontier experience on culture. We focus on the subset of their main results for which replication data is publicly available. These are columns 1, 2, 4, 6, and 7 of their table 3. Panel A in our Table (ref) replicates those results. From columns (1)--(3) we see that individuals who live in counties with more exposure to the frontier prefer cutting spending on the poor, on welfare, and to reduce debt by spending cuts. Moreover, these point estimates are statistically significant at conventional levels. From columns (4) and (5), we see that counties with more exposure to the frontier have lower property taxes and are more likely to vote for Republicans. As BFG2020 argue, these baseline results support the conjecture that frontier life led to opposition to government intervention and redistribution.
The baseline results in Table 1 rely on a selection on observables assumption, that treatment $X$ is exogenous after adjusting for the observed covariates $(W_0,W_1)$. How plausible is this assumption? BFG2020 say
Their first two approaches continue to rely on selection on observables, and consist of including additional control variables. We focus on their third strategy: to use a formal econometric method to assess the importance of omitted variables.
BFG2020 use Oster's Oster2019 to assess the impact of omitted variables on their results. That method uses a sensitivity parameter $\delta_\text{resid}$, which is informally described in Oster's appendix A (see appendix C of our companion paper DiegertMastenPoirier2025 for a detailed, formal description of this parameter). The variables $(X,W_1,W_2)$ are replaced by $(X^{\perp W_0}, W_1^{\perp W_0}, W_2^{\perp W_0})$ to adjust for the non-calibration covariates $W_0$. Finally, the auxiliary sensitivity parameter in Oster's analysis, $R_{Y \sim X,W_1,W_2 \mathrel{\mathsmaller{\bullet}} W_0}^2$, is set to $1.3 \cdot \widehat{R}_{Y \sim X,W_1 \mathrel{\mathsmaller{\bullet}} W_0}^2$. Using these parameters and Oster's analysis, BFG2020 report the estimated breakdown point, which is a measure of how large $\delta_\text{resid}$ must be for the estimated effect to be fully explained away by the unobservables.
The second row of Panel B of Table (ref) shows sample analog estimates of this breakdown point, which is commonly referred to as Oster's delta. The first row of Panel B shows the values of Oster's delta as reported in table 2 of BFG2020. These were computed incorrectly.\footnote{There are three differences between the two rows in Panel B. First, it appears to us that, rather than using the correct expression in Proposition 3 of Oster2019, BFG2020 set the first displayed equation on page 193 of Oster2019 equal to zero and solved for $\delta_\text{resid}$. That does not give the correct breakdown point. This mistake was not unique to those authors; while replicating other results, we discovered the same mistake in several other papers published in `top 5' economics journals. Second, BFG2020 used $\widehat{R}_{Y \sim X,W_0,W_1}^2$ and $\widehat{R}_{Y \sim X,W_0}^2$ to adjust for $W_0$ rather than $\widehat{R}_{Y \sim X,W_1 \mathrel{\mathsmaller{\bullet}} W_0}^2$ and $\widehat{R}_{Y \sim X \mathrel{\mathsmaller{\bullet}} W_0}^2$. Third, as we discussed earlier, they include the individual demographics and survey wave fixed effects in $W_1$ rather than $W_0$. This is a valid choice, but we chose to instead include these variables in $W_0$ to keep the covariates in $W_1$ constant across columns (1)--(5) for comparability.} Based on these estimates, BFG2020 conclude:
This conclusion remains unchanged if the same robustness cutoff is applied to the correctly computed $\delta_\text{resid}^\text{bp}$ estimates.
In our companion paper DiegertMastenPoirier2025, we argue that robustness conclusions based on $\delta_\text{resid}$ are particularly problematic when the controls are endogenous. Are the controls in this application plausibly exogenous, however? The answer depends on which omitted variables $W_2$ we are concerned about. BFG2020 does not specifically describe the unmeasured omitted variables of concern, nor do they discuss the plausibility of exogenous controls. However, in their extra robustness checks they consider the variables listed in Table (ref).
The additional omitted variables of concern might therefore be similar to these variables. Thus the question is: Are all of the geographic/climate variables in $W_1$ uncorrelated with variables like these? This seems unlikely, especially since many of these additional variables are also geographic/climate type variables. Moreover, although this assumption is not falsifiable---since $W_2$ is unobserved---we can assess its plausibility by examining the correlation structure of the observed covariates. Specifically, we estimate the values $R_{W_{1k} \sim W_{1,-k} \mathrel{\mathsmaller{\bullet}} W_0}^2$ for each covariate $k$ in $W_1$. For a given $k$, this is the population R-squared from the regression of $W_{1k}$ on the rest of the calibration covariates $W_{1,-k}$, after partialling out the control covariates $W_0$. We described these values in section (ref). Table (ref) shows sample analog estimates of these values.
The estimates in Table (ref) show a substantial range of correlation between the observed covariates in $W_1$. Recall that the exogenous controls assumption says that each element of $W_1$ is uncorrelated with $W_2$, after partialling out $W_0$. So under that assumption, if $W_2$ was included in this table, it would have a value of zero. Therefore, if $W_2$ is a variable similar to the components of $W_1$ then we would expect exogenous controls to fail. This suggests that, in this application, robustness conclusions based on $\delta_\text{resid}$ could be incorrect. Consequently, one should instead use sensitivity analysis methods which are valid even when the controls are endogenous.
Next we present the findings from the sensitivity analysis that we developed in section (ref). We first discuss our two main robustness summary statistics, and then we give a more detailed analysis.
Robustness Summary Statistics. Our simplest result, Theorem (ref), only uses a single sensitivity parameter $\bar{r}_X$. The first row of Panel C of Table (ref) reports sample analog estimates of the breakdown point $\bar{r}_X^\text{bp}$ described in Corollary (ref). This is the largest amount of selection on unobservables, as a percentage of selection on observables, allowed for until we can no longer conclude that $\beta_\text{long}$ is nonzero. Recall that, since this result allows for arbitrarily endogenous controls, Theorem (ref) implies that $\bar{r}_X^\text{bp} < 1$. This does not imply that these results should always be considered non-robust. Instead, when the calibration covariates $W_1$ are a set of variables that are important for treatment selection, researchers should consider large values of $\bar{r}_X^\text{bp}$ to indicate the robustness of their baseline results. For example, in columns (4) and (5) of Panel C we see that the breakdown point estimates for the two behavior based outcomes are 72.6% and 80.4%. For example, for the average Republication vote share outcome, we can conclude $\beta_\text{long} > 0$ as long as selection on unobservables is at most 80.4% as large as selection on observables. In contrast, the breakdown point estimates in columns (1)--(3) are substantially smaller: between about 3% and 6%. For these outcomes, we therefore only need selection on unobservables to be at least 3 to 6% as large as selection on observables to overturn our conclusion that $\beta_\text{long} > 0$. Thus, without imposing restrictions on the impact of unobservables on outcomes or on the magnitude of control endogeneity, we find that the analysis using questionnaire based outcomes is highly sensitive to selection on unobservables. In contrast, the analysis using behavior based outcomes is quite robust to selection on unobservables.
The breakdown point $\bar{r}_X^\text{bp}$ is a conservative measure of robustness to omitted variables, because it does not restrict the impact of the omitted variables on outcomes. A less conservative measure of robustness is $\bar{r}^\text{bp}$, which restricts the omitted variables' impact on treatment and on outcomes. The second row of Panel C of Table (ref) reports sample analog estimates of this breakdown point. In columns (4) and (5) we see that the breakdown point increases to above 90%. For example, for the average Republican vote share outcome, we can conclude $\beta_\text{long} > 0$ as long as selection on unobservables is at most 96% as large as selection on observables, and as long as the impact of omitted variables on outcomes is at most 96% as large as the impact of the observables on outcomes. The breakdown point estimates in columns (1)--(3) are smaller, between 54% and 84%. Nonetheless, these breakdown points are substantially larger than the corresponding estimates of $\bar{r}_X^\text{bp}$. For example, for the cut spending on the poor outcome, the breakdown point increases from about 3% to 54% once we restrict the impact of omitted variables on outcomes. Overall, this less conservative measure of robustness shows that it is possible to conclude that the analysis using questionnaire based outcomes is robust. But since there is more variation in the breakdown points for columns (1)--(3), this robustness conclusion depends more importantly on one's subjective judgment about the cutoff for robustness, unlike the analysis for behavior based outcomes.
Calibrating $\bar{r}_X$. Table (ref) shows estimates of the calibration parameters $\rho_k$ that we discussed in section (ref). Although the treatment variable is the same across all specifications in Table (ref), the estimation datasets vary. Consequently, the estimates of $\rho_k$ vary across specifications as well. Here we only show the estimates corresponding to the specifications in columns (1) and (5) for brevity. First consider column (5), the results for average Republican vote share. Here we see that $\widehat{\rho}_k$ is smaller than the $\bar{r}_X^\text{bp}$ breakdown point estimate of 80.4% for all but one covariate value. This suggests that the results for average Republican vote share is quite robust to selection on unobservables, assuming we view the calibration parameter estimates as plausible values of the unknown ratio $r_X$. In contrast, consider column (1), the results for the cut spending on the poor outcome variable. In this case, the $\bar{r}_X^\text{bp}$ breakdown point estimate of 2.81% is smaller than all the calibration parameters $\widehat{\rho}_k$. Even if we use the less conservative breakdown point $\widehat{\bar{r}}^\text{bp} = 54.1$%, this value is still smaller than half of the calibration parameters. This suggests that the results for cut spending on the poor are sensitive to selection on unobservables, again assuming the calibration parameter estimates are plausible values for the unknown $r_X$. Overall, our calibration analysis confirms the findings discussed above: The conclusions using the behavior based outcomes are substantially more robust than the conclusions using questionnaire based outcomes.
Beyond the Summary Statistics. Our results also allow researchers to explore the impact of a wide range of assumptions on the omitted variables. We present this analysis next. For brevity we discuss just one questionnaire based outcome, cut spending on the poor, and one behavior based outcome, average Republican vote share. Figure (ref) shows the results for average Republican vote share. The solid lines in the left plot show the estimated identified set for $\beta_\text{long}$ as a function of $\bar{r}_X$, allowing for arbitrarily endogenous controls and no restrictions on the outcome equation. This is the set described by Theorem (ref). The horizontal intercept is the estimated breakdown point $\widehat{\bar{r}}_X^\text{ bp} = 80.4\%$, as reported in Panel C, column (5) of Table (ref). The dashed lines show the estimated identified set once we impose the common maximal impact assumption $\bar{r}_Y = \bar{r}_X$. The horizontal intercept is the estimated breakdown point $\widehat{r}^\text{ bp} = 96\%$. For all values of $\bar{r}_X$, we see that imposing the restriction $\bar{r}_Y = \bar{r}_X$ shrinks the size of the identified set substantially. The figure also shows the estimated calibration parameters from Table (ref) as tick marks on the horizontal line at zero.
Next we consider robustness to choices of $\bar{r}_Y \neq \bar{r}_X$. To show this, we plot the estimated breakdown frontier $\bar{r}_Y^\text{bf}(\bar{r}_X, 0)$ that we showed how to compute in Theorem (ref). This is shown as the solid black line in the right plot of figure (ref). For any value of $(\bar{r}_X, \bar{r}_Y)$ below this line we can conclude that $\beta_\text{long} > 0$. For example, $(\bar{r}_X, \bar{r}_Y) = (2, 0.5)$ lies below this line. This means that our conclusion that $\beta_\text{long} > 0$ is robust to unobservables up to twice as strong as the observables for their impact on treatment, and up to half as strong as the observables for their impact on outcomes.
In section (ref) we also extended our results to allow researchers to restrict the magnitude of control endogeneity as well. These restrictions shift the breakdown frontier away from zero, as shown by the gray lines in the right plot of figure (ref). These plots correspond to the restriction $R_{W_2 \sim W_1 \mathrel{\mathsmaller{\bullet}} W_0} \in [0,\bar{c}]$ for the various displayed values of $\bar{c}$. For example, if we set $\bar{c} = 0.7$ then the point $\bar{r}_X = \bar{r}_Y = 110\%$ is now below the breakdown frontier. Hence with this restriction on control endogeneity, our conclusion $\beta_\text{long} > 0$ is robust to unobservables up to 1.1 times as strong as unobservables in for their impact on treatment and on outcomes. Note that $0.7^2 = 0.49$ is around the middle of the distribution of values in Table (ref), and hence might be considered a moderate or slightly conservative value of the magnitude of control endogeneity. If we impose exogenous controls ($\bar{c} = 0$) then we can allow the impact of the omitted variable on outcomes to be 200% as large as the observables and the impact of the omitted variable on treatment to be up to about 240% as large as the observables, and yet still conclude that $\beta_\text{long} > 0$.
These findings suggest that the empirical conclusions for average Republican vote share are quite robust to failures of the selection on observables assumption. In contrast, we next consider the analysis for the cut spending on the poor outcome. Figure (ref) shows the results. The solid lines in the left plot show the estimated identified sets for $\beta_\text{long}$ as a function of $\bar{r}_X$ on the horizontal axis. The horizontal intercept gives an estimated value for $\bar{r}_X^\text{bp}$ of 2.81%, as reported in Panel C, column (1) of Table (ref). The dashed lines show the estimated identified sets once we impose the common maximal impact assumption $\bar{r}_Y = \bar{r}_X$. This assumption substantially shrinks the identified set, and increases the estimated breakdown point up to 54.1%. This point can also be seen as the intersection between a 45 degree line and the solid black line on the right plot of figure (ref), which shows the estimated breakdown frontier $\bar{r}_Y^\text{bf}(\bar{r}_X, 0)$. Compared to figure (ref), these breakdown frontiers are all shifted substantially toward the origin, suggesting that the conclusion $\beta_\text{long} > 0$ is relatively less robust for this outcome variable.
Overall, there are some cases where the results for cutting spending on the poor could be considered robust, especially if one is willing to restrict the impact of omitted variables on outcomes. But there are also cases where the results could be considered sensitive. In contrast, the results for average Republican vote share are robust across a wide range of relaxations of the baseline model. Similar findings hold for the other three outcome variables: The three results using the questionnaire based outcomes tend to be more sensitive to assumptions on the unobservables than the two results using behavior based outcomes.
Overall, a sensitivity analysis based on our new methods leads to a richer set of empirical conclusions about the robustness of the long run effect of the frontier experience on modern culture than those originally obtained by BFG2020. We found that their analysis using questionnaire based outcomes is generally more sensitive to the presence of omitted variables than their analysis using property tax levels and voting patterns. This has several empirical implications.
First, the questionnaire based outcomes are the most easily interpretable as measures of opposition to redistribution, regulation, and preferences for small government. In contrast, it is less clear that property taxes and Republican presidential vote share alone should be interpreted as direct measures of opposition to redistribution. So the fact that the questionnaire based outcomes are more sensitive to the presence of omitted variables suggests that Bazzi et al.'s overall conclusion in support of the “frontier thesis” might be considered more tentative than previously stated. Second, it suggests that the impact of frontier life may occur primarily through broader behavior based channels like elections, rather than individuals' more specific policy preferences and behavior in their personal lives. It may be useful to explore this difference in future empirical work.
Finally, note that BFG2020 perform a wide variety of additional supporting analyses that we have not examined here. In particular, their figure 5 considers another set of outcome variables: Republican vote share in each election from 1900 to 2016. In contrast, our analysis above looked only at one election outcome: the average Republican vote share over the five elections from 2000 to 2016. They use these additional baseline estimates along with a qualitative discussion of the evolution of Republican party policies over time to argue that the average Republican vote share outcome between 2000--2016 can be interpreted as a measure of opposition to redistribution. It would be interesting to also apply our methods to these additional analyses.
This paper develops a new method to assess the sensitivity of regression results to omitted variables that does not require assuming the controls are exogenous. Our approach maintains a key aspect of the popular methods of AltonjiElderTaber2005 and Oster2019 which let researchers calibrate the magnitude of the sensitivity parameter by comparing a measure of selection on observables with a measure of selection on unobservables, while simultaneously resolving several limitations of those earlier methods. Our results are also simple to implement in practice, via the accompanying Stata package regsensitivity. Finally, in our empirical application to Bazzi et al.'s BFG2020 study of the impact of frontier life on modern culture, we showed that allowing for endogenous controls does matter in practice, leading to richer empirical conclusions than those obtained in BFG2020.