Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
72,354 characters · 2 sections · 83 citation commands
How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice Based on 67 Replicated Studies
\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }{Introduction}
The instrumental variable (IV) approach is a widely used empirical method in the social sciences, including political science, for establishing causal relationships. It is often used when selection on observables is implausible, experimentation is infeasible or unethical, and rule-based assignments that allow for sharp regression-discontinuity (RD) designs are unavailable. In recent years, there has been a growing number of articles published in top political science journals, such as the American Political Science Review (APSR), American Journal of Political Science (AJPS), and Journal of Politics (JOP), that use IV as a primary causal identification strategy. This trend can be traced back to the publication of Mostly Harmless Econometrics angrist2008mostly, which popularized the modern interpretation of IV designs, and sovey2011instrumental, which clarifies the assumptions required by an IV design and provides a useful checklist for political scientists.
Despite its popularity, the IV approach has faced scrutiny from researchers who note that two-stage least-squares (2SLS) estimates are often much larger in magnitude than “naïve” ordinary-least-squares (OLS) estimates, even when the main concern with the latter is upward omitted-variables bias.\footnote{For example, in the 2016 National Bureau of Economic Research--Political Economy Meeting, following a presentation of a study using an IV approach, the late political economist Alberto Alesina asked the audience: “How come 2SLS estimates are always five times bigger than OLS estimates in political economy?” We dedicate this paper to him for his seminal contributions to the field of political economy.} Others have raised concerns about the validity of the commonly used inferential method for 2SLS estimation Lee2020-mi,Young2022.
These observations motivate our systematic examination of the use of IVs in the empirical political science literature. We set out to replicate all studies published in the APSR, AJPS, and JOP during the past thirteen years (2010-2022) that use an IV design with a single endogenous variable as one of the main identification strategies.\footnote{Focusing on design with a single endogenous variable allows us to calculate the first-stage correlation coefficient (or $R^2$) and apply tools such as the Anderson-Rubin (AR) test and the $tF$ test (when there is only a single instrument). Moreover, we find it difficult to justify the exclusion restriction in a multiple-treatment-multiple-instrument setting in the first place.} Out of 114 articles meeting this criterion, 71 have complete replication materials online, which is concerning in itself. We successfully replicated at least one primary IV result in 67 out of the remaining 71 articles. Among the 67 articles, three articles feature two distinct IV designs, each yielding two separate replicable IV results.
Using data from these 70 IV designs, we conduct a programmatic replication exercise and find three troubling patterns. First, a significant number of IV designs in political science overestimate the first-stage partial $F$-statistic by failing to adjust standard errors (SEs) for factors such as heteroskedasticity, serial correlation, or clustering structure. Using the effective $F$-statistic Olea2013-pa, we find that at least 11% of the published IV studies rely on what econometricians call “weak instruments,” the consequences of which have been well-documented in the literature. See andrews2019weak for a comprehensive review.
Second, obtaining valid statistical inferences for IV estimates remains challenging. Almost all studies we have replicated rely on $t$-tests for the 2SLS estimates based on analytic SEs and traditional critical values (such as 1.96 for statistical significance at the 5% level). Using analytic SEs, IV estimates are already shown to be much more imprecise than OLS estimates. When employing bootstrapping procedures, the AR test, or the $tF$ procedure---an $F$-statistic-dependent $t$-test Lee2020-mi---for hypothesis testing, we find that 17-35% of the designs cannot reject the null hypothesis of no effect at the 5% level. In contrast, only 10% of studies based on originally reported SEs or $p$-values fail to reject the null hypothesis. This discrepancy suggests that many studies may have underestimated the uncertainties associated with their 2SLS estimates.
What is even more concerning is that an IV approach can produce larger biases than OLS when weak instruments amplify biases due to failures of IVs' unconfoundedness or exclusion restrictions. We observe that in 68 out of the 70 designs (97%), the 2SLS estimates have a larger magnitude than the naïve OLS estimates obtained from regressing the outcome on potentially endogenous treatment variables and covariates; 24 of these (34%) are at least five times larger. This starkly contrasts with the common rationale for using IV, which is to mitigate the upward bias in treatment effect estimates from OLS. Moreover, we find the ratio between the magnitudes of the 2SLS and OLS estimates is strongly negatively correlated with the strength of the first stage among studies that use non-experimental instruments, and the relationship is almost nonexistent among experimental studies. While factors such as heterogeneous treatment effects and measurement error might be at play, we contend that this phenomenon primarily stems from a combination of weak instruments and the failure of unconfoundedness or the exclusion restriction. Intuitively, because the 2SLS estimator is a ratio, an inflated numerator from invalid instruments paired with a small denominator due to a weak first stage leads to a disproportionately large estimate. Publication bias and selective reporting exacerbate this issue.
What do these findings imply for IV studies in political science? First, the traditional $F$-tests for IV strength, particularly when using classic analytic SEs, often mask the presence of weak instruments. Second, when operating with these weak instruments, particularly in over-identified scenarios, traditional $t$-tests do not adequately represent the considerable uncertainty surrounding the 2SLS estimates, paving the way for selective reporting and publication bias. Last but not least, many 2SLS estimates likely bear significant biases due to violations of unconfoundedness or the exclusion restriction, and weak instruments further exacerbate these biases. While we cannot pinpoint exactly which estimates are problematic, these issues seem to be pervasive across observational IV studies. The objective of this paper, however, is not to discredit existing IV research or dissuade scholars from using IVs. On the contrary, our intent is to caution researchers against the pitfalls of ad-hoc justifications for IVs in observational research and provide constructive recommendations for future practices. These suggestions include accurately quantifying the strength of instruments, conducting valid inference for IV estimates, as well as implementing additional validation exercises, such as placebo tests, to bolster the identifying assumptions.
Our work builds on a growing literature evaluating IV strategies in social sciences and offering methods to improve empirical practice. Notable studies include Young2022, which finds IV estimates to be more sensitive to outliers and conventional $t$-tests to understate uncertainties; jiang2017have, which observes larger IV estimates in finance journals and attributed this to exclusion restriction violations and weak instruments; mellon2020rain, which emphasizes the vulnerability of weather instruments; dieterle2016simple, which develops a quadratic over-identification test and discovers significant non-linearities in the first stage regression; FeltonStewart2022, which finds unstated assumptions and a lack of weak-instrument robust tests in top sociology journals; and cinelli2022omitted, which proposes a sensitivity analysis for IV designs in an omitted variable bias framework. This study is the first comprehensive replication effort focusing on IV designs in political science and uses data to shed light on the consequences of weak instruments interacting with failures of unconfoundedness or the exclusion restriction.
\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }{Theoretical Refresher}
In this section, we offer a brief overview of the IV approach, including the setup, the key assumptions, and the 2SLS estimator. We then discuss potential pitfalls and survey several inferential methods. To cover the vast majority of IV studies in political science, we adopt a traditional constant treatment effect approach, which imposes a set of parametric assumptions. For example, of our replication sample, 51 designs (73%) employ continuous treatment variables, and 49 (70%) use continuous IVs. Most of these studies make no reference to treatment effect heterogeneity and are ill-suited for the local average treatment effect framework angrist1996identification.
Apart from the canonical use of IVs in addressing non-compliance in experimental studies, we observe that in the majority of the articles we review, researchers use IVs to establish causality between a single treatment variable $d$ and an outcome variable $y$ in observational settings. The basic idea of this approach is to use a vector of instruments $z$ to isolate “exogenous” variation in $d$ (i.e., the variation in $d$ that is not related to potential confounders) and estimate its causal effect on $y$. For simplicity, we choose not to include any additional exogenous covariates in our discussion. This is without loss of generality because, by the Frisch-Waugh-Lovell theorem, we can remove these variables by performing a regression of $y$, $d$, and each component of $z$ on the controls and then proceeding with our analysis using the residuals instead.
Imposing a set of parametric assumptions, we define a system of simultaneous equations:
in which $y$ is the outcome variable; $d$ is a scalar treatment variable; $z$ is a vector of instruments for $d$; and $\tau$ captures the (constant) treatment effect and is the key quantity of interest. The error terms $\varepsilon$ and $\nu$ may be correlated. The endogeneity problem for $\tau$ in Equation ((ref)) arises when $d$ and $\varepsilon$ are correlated, which renders $\hat\tau_{OLS}$ from a naïve OLS regression of $y$ on $d$ inconsistent. This may be due to several reasons: (1) unmeasured omitted variables correlated with both $y$ and $d$; (2) measurement error in $d$, or (3) simultaneity or reverse causality, which means $y$ may also affect $d$. Substituting $d$ in Equation ((ref)) using Equation ((ref)), we have the reduced form equation:
Substitution establishes that $\gamma = \tau \pi$, rearranging yields $\tau = \frac{\gamma}{\pi}$ (assuming a single instrument, but the intuition carries over to cases with multiple instruments). The IV estimate, therefore, is the ratio of the reduced-form and first-stage coefficients. To identify $\tau$, we make the following assumptions greene2003econometric.
Conceptually, unconfoundedness and the exclusion restriction are two distinct assumptions and should be justified separately in a research design. However, because violations of either assumption lead to the failure of the 2SLS moment condition, $\mathbb{E}[z\varepsilon] = 0$, and produce observationally equivalent outcomes, we consider both to be integral components of Assumption 2.
Under Assumptions (ref) and (ref), the 2SLS estimator is shown to be consistent for the structural parameter $\tau$. Consider a sample of $N$ observations. We can write $\mathbf{d} = (d_1, d_2, \cdots, d_N)'$ and $\mathbf{y}= (y_1, y_2, \cdots, y_N)'$ as $(N\times 1)$ vectors of the treatment and outcome data, and $\mathbf{z}= (z_1, z_2, \cdots, z_N)'$ as an $(N\times p_z)$ matrix of instruments in which $p_z$ is the number of instruments. The 2SLS estimator is written as follows:
in which $\mathbf{P}_{z} = \mathbf{z} \left({\mathbf{z}' \mathbf{z}} \right)^{-1} \mathbf{z}'$ is the hat-maker matrix from the first stage which projects the endogenous treatment variable $\mathbf{d}$ into the column space of $\mathbf{z}$, thereby in expectation preserving only the exogenous variation in $\mathbf{d}$ that is uncorrelated with $\varepsilon$. This formula permits the use of multiple instruments, in which case the model is said to be “overidentified.” The 2SLS estimator belongs to a class of generalized method of moments (GMM) estimators taking advantage of the moment condition $\mathbb{E}[z\varepsilon] =0$, including the two-step GMM hansen1982large and limited information maximum likelihood (LIML) estimators anderson1982evaluation. We use the 2SLS estimator throughout the replication exercise because of its simplicity and because every single paper in our replication sample uses it in at least one specification.
When the model is exactly identified, i.e., the number of treatment variables equals the number of instruments, the 2SLS estimator can be simplified as the IV estimator: $\hat{\tau}_{\text{2SLS}} = \hat{\tau}_{\text{IV}} = \left({\mathbf{z}'\mathbf{d}} \right)^{-1} \mathbf{z}'\mathbf{y}$. In the case of one instrument and one treatment, the 2SLS estimator can also be written as a ratio of two sample covariances: $\hat{\tau}_{2SLS} = \hat{\tau}_{IV} = \frac{\hat\gamma}{\hat\pi} = \frac{\widehat{\rm Cov}(y, z)}{\widehat{\rm Cov}(d, z)}$, which illustrates that the 2SLS estimator is a ratio between reduced-form and first-stage coefficients in this special case. This further simplifies to a ratio of the differences in means when $z$ is binary, which is called a Wald estimator.
The challenges with 2SLS estimation and inference are mostly due to violations of Assumptions (ref) and (ref). Such violations can result in (1) significant uncertainties around 2SLS estimates and size distortion for $t$-tests due to weak instruments even when the exogeneity assumption is satisfied; and (2) potentially larger biases in 2SLS estimates compared to OLS estimates when both assumptions are violated.
\@startsection{paragraph}{4} \z@\z@{-\fontdimen2\font} *{Inferential problem due to weak instruments.} Since the IV coefficient is a ratio, the weak instrument problem is a “divide-by-zero” problem, which arises when $\text{Cov}(z, d) \approx 0$ (i.e., when the relevance assumption is violated). The instability of ratio estimators like $\wh\tau_{\text{2SLS}}$ when the denominator is approximately zero has been extensively studied going back to fieller1954some. The conventional wisdom in the past two decades has been that the first-stage partial $F$-statistic needs to be bigger than 10, and it should be clearly reported Staiger1997-lo. The cutoff, as a rule of thumb, is chosen based on simulation results to meet two criteria under i.i.d. errors: (1) in the worst case, the bias of the 2SLS estimator does not exceed 10% of the bias of the OLS estimator, and (2) a $t$-test based on the 2SLS estimator with a size of 5% does not lead to size over 15%. These problems are further exacerbated in settings where units belong to clusters with strong within-cluster correlation, where a small number of observations or clusters may heavily influence estimated results Young2022. Recently, however, angrist2023one argue that the conventional inference strategies are reliable in just-identified settings with independent errors. The weak instrument issue is indeed most concerning in heavily over-identified scenarios.
The literature has discussed at least three issues caused by weak instruments when the exogeneity assumption is satisfied. First, under i.i.d. errors, a weak first stage exacerbates the finite-sample bias of the 2SLS estimator toward the inconsistent OLS estimator, thereby reproducing the endogeneity problem that an IV design was meant to solve Staiger1997-lo. Additionally, when the first stage is weak, the 2SLS estimator may not have a mean; its median is centered around the OLS coefficient hirano2015location. Second, the 2SLS estimates become very imprecise. To illustrate, a commonly used variance estimator for $\hat{\tau}_{IV}$ is $\hat{\mathbb{V}}(\hat{\tau}_{IV}) \approx \hat{\sigma}^2/(\ensuremath{\sum_{i=1}^N} (d_i - \overline{d})^2 R_{dz}^2) = \hat{\mathbb{V}}(\hat{\tau}_{OLS}) /R_{dz}^2$, in which $\hat{\sigma}^2$ is a variance estimator for the error term and $R^2_{dz}$ is the first stage $R^2$. $\hat{\mathbb{V}}(\hat{\tau}_{IV})$ is generally larger than $\hat{\mathbb{V}}(\hat{\tau}_{OLS})$ and increasing in $1/R^2_{dz}$. A third and related issue is that the $t$-tests are of the wrong size and the $t$-statistics do not follow a $t$-distribution nelson1990some. This is because the distribution of $\hat{\tau}$ is derived from its linear approximation of $\hat{\tau}$ in ($\hat{\gamma}, \hat{\pi}$), wherein normality of the two OLS coefficients implies the normality of their ratio. However, this normal approximation breaks down when $\hat{\pi} \approx 0$. Moreover, this approximation failure cannot generally be rectified by bootstrapping andrews2009validity; Young2022 argues that it nevertheless allows for improved inference when outliers are present. Overall, valid IV inference relies crucially on strong IVs.
Generally, there are two approaches to conducting inference in an IV design: pretesting and direct testing. The pretesting approach involves using an $F$-statistic to test the first stage strength, and if it exceeds a certain threshold (e.g., $F > 10$), proceeding to test the null hypothesis about the treatment effect (e.g., $\tau = 0$). Nearly all reviewed studies employ this approach. The direct testing approach, in contrast, does not rely on passing a pretest. We examine four inferential methods for IV designs, with the first three related to pretesting and the last one being a direct test.
First, Olea2013-pa propose the effective $F$-statistic for both just-identified and over-identified settings and accommodates robust or cluster-robust SEs. The effective $F$ is a scaled version of the first-stage $F$-statistic and is computed as $F_{\text{Eff}} = \hat{\pi}'\hat{Q}_{\text{ZZ}} \hat{\pi} / \text{tr}(\hat{\Sigma}_{\pi \pi} \hat{Q}_{\text{ZZ}})$, where $\hat{\Sigma}_{\pi \pi}$ is the variance-covariance matrix of the first stage regression, and $\hat{Q}_{\text{ZZ}} = \frac{1}{N} \ensuremath{\sum_{i=1}^N} z_i z_i'$. In just-identified cases, $F_{\text{Eff}}$ is the same as an $F$-statistic based on robust or cluster-robust SEs. The authors derive the critical values for $F_{\text{Eff}}$ and note that the statistic and corresponding critical values are identical to the better-known robust $F$-statistic $\hat{\pi} \hat{\Sigma}_{\pi \pi}^{-1} \hat{\pi}$ and corresponding stock2005asymptotic critical values. $F_{\text{Eff}}>10$ is shown to be a reasonable rule of thumb under heteroskedasticity in simulations Olea2013-pa, andrews2019weak.
Second, Young2022 recommends researchers report two types of bootstrap confidence intervals (CIs), bootstrap-c and bootstrap-t, for $\hat{\tau}_{2SLS}$ under non-i.i.d. errors with outliers, which is common in social science settings. They involve $B$ replications of the following procedure: (1) sample $n$ triplets $(y_i^*, d_i^*, z_i^*)$ independently and with replacement from the original sample (with appropriate modifications for clustered dependence) and (2) on each replication, compute the 2SLS coefficient and SE, as well as the corresponding test statistic $t^* = \hat{\tau}^*_{\text{2SLS}} / \hat{\text{SE}} (\hat{\tau}^*_{\text{2SLS}})$. The bootstrap-c method calculates the CIs by taking the $\alpha/2$ and $(1-\alpha/2)$ percentiles of the bootstrapped 2SLS coefficients $\hat{\tau}_{\text{2SLS}}^*$, while the bootstrap-t method calculates the percentile-$t$ refined CIs by plugging in the $\alpha/2$ and $(1-\alpha/2)$ percentile of the bootstrapped $t$ statistics into the expression $\hat{\tau}_{\text{2SLS}} \pm t^*_{\alpha \mid 1 - \alpha} \hat{\text{SE}}(\hat{\tau}^*_{\text{2SLS}})$. hall1996bootstrap show that bootstrap-t achieves an asymptotic refinement over bootstrap-c. Note that $t$-tests based on bootstrapped SEs may be overly conservative Hahn2021-lq and, hence, are not recommended.
Third, in just-identified single treatment settings, Lee2020-mi propose the $tF$ procedure that smoothly adjusts the $t-$ratio inference based on the first-stage $F$-statistic, which improves upon the ad-hoc screening rule of $F > 10$. The adjustment factor applied to 2SLS SEs is based on the first stage $t-$ratio $\hat{f} \vcentcolon= \hat{\pi}/\sqrt{\hat{\mathbb{V}}(\hat{\pi})}$, with the first stage $\hat{F} = \hat{f}^2$, and relies on the fact that the distortion from employing the standard 2SLS $t$-ratio $\hat{t} \vcentcolon= \hat{\tau}/\sqrt{\hat{\mathbb{V}}(\hat{\tau})}$ can be quantified in terms of an $\hat{F}$ statistic, which gives rise to a set of critical values for a given pair of $\hat{t}$ and $\hat{F}$. The authors also show that if no adjustment is made to the $t$-test's critical value (e.g., using 1.96 as the threshold for 5% statistical significance), a first stage $\hat{F}$ of 104.7 is required to guarantee a correct size of $5\%$ for a two-sided $t$-test for the 2SLS coefficient.
Finally, where there is one endogenous treatment variable, the AR procedure, which is essentially an $F$-test on the reduced form, is a direct inferential method robust to weak instruments anderson1949estimation, chernozhukov2008reduced. Without loss of generality, assume that we are interested in testing the null hypothesis that $\tau = 0$, which then implies that the reduced form coefficient from regressing $y$ on $z$ is zero, i.e., $\gamma = 0$. This motivates the following procedure: given a set $\mathcal{T}$ of potential values for $\widetilde{\tau}$, for each value $\widetilde{\tau}$, construct $\widetilde{y} = y - d \widetilde{\tau}$, and regress $\widetilde{y}$ on $z$ to obtain a point estimate $\widetilde{\gamma}$ and (robust, or cluster robust) covariance matrix $\widetilde{\mathbb{V}}(\widetilde{\gamma})$, and construct a Wald statistic $\widetilde{W}_s(\widetilde{\gamma}) \vcentcolon= \widetilde{\gamma}' \widetilde{\mathbb{V}}(\widetilde{\gamma})^{-1} \widetilde{\gamma}$. Then, the AR CI (or confidence set) is the set of $\widetilde{\gamma}$ such that $\widetilde{W}_s(\widetilde{\gamma}) \leq c(1-p)$ where $c(1-p)$ is the $(1-p)^{\text{th}}$ percentile of the $\chi^2_1$ distribution. The AR test requires no pretesting and is shown to be the uniformly most powerful unbiased test in the just-identified case moreira2009tests. It is less commonly used than pretesting procedures possibly because researchers are more accustomed to using $t$-tests than $F$-tests and reporting SEs rather than CIs. A potential limitation of the AR test is that its CIs can sometimes be empty or disconnected, and therefore lack a Bayesian interpretation under uninformative priors.\footnote{We thank Guido Imbens for highlighting this point.}
\@startsection{paragraph}{4} \z@\z@{-\fontdimen2\font} *{Bias amplification and the failure of the exogeneity assumption.} When the number of instruments is bigger than the number of endogenous treatments, researchers can use an over-identification test to gauge the plausibility of Assumption (ref), the exogeneity assumption arellano2002sargan. However, such a test is often underpowered and has bad finite sample properties davidson2015bootstrap. In just-identified cases, Assumption (ref) is not directly testable. When combined with weak instruments, even small violations of unconfoundedness or the exclusion restriction can produce inconsistency. This is because: $\text{plim} \;\hat{\tau}_{IV} = \tau + \frac{\text{Cov}(z, \varepsilon)}{\text{Cov}(z, d)}$. When $\text{Cov}(z, d) \approx 0$, even small violations of exogeneity, i.e., $\text{Cov}(z, \varepsilon) \neq 0$, will enlarge the second term, resulting in large biases. Thus, the two identifying assumption failures exacerbate each other: having weak instruments compounds problems from confounding or exclusion restriction violations, and vice versa. With invalid instruments, it is likely that the asymptotic bias of the 2SLS estimator is much greater than that of the OLS estimator, i.e., $\left|\frac{\text{Cov}(z, \varepsilon)}{\text{Cov}(z, d)}\right| \gg \left|\frac{\text{Cov}(d, \varepsilon)}{\mathbb{V}\left[ d\right]}\right|$ in the single instrument case.\footnote{We are not the first to make this argument. According to hahn2005estimation: “[T]he empirical finding that the 2SLS estimate increases compared to the OLS estimate may indicate that the instrument is not orthogonal to the stochastic disturbance. The resulting bias can be substantial. Indeed, it could exceed the OLS bias, leading to an increase in the estimated 2SLS coefficient over the estimated OLS coefficient.”}
While the inferential problem can be alleviated by employing alternative inferential methods as described above, addressing violations of unconfoundedness or the exclusion restriction is more challenging since it is fundamentally a research design issue that should be tackled at the design stage. Researchers often devote significant effort to arguing for unconfoundedness and exclusion restrictions in their settings. In Section A3 of the SM, we provide an exposition of the zero-first-stage (ZFS) test bound2000compulsory, which is essentially a placebo test on a subsample where the instrument is expected to be uncorrelated with the treatment, to help researchers gauge the validity of their instruments. These estimates can then be used to debias the 2SLS estimate using the methods proposed in Conley2012-mu.
\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }{Data and Types of Instruments}
In this section, we first discuss our case selection criteria and replication sample, which is the focus of our subsequent analysis. We then describe the types of instruments in the replicable studies.
\@startsection{paragraph}{4} \z@\z@{-\fontdimen2\font} *{Data.} We examine all empirical articles published in the APSR, AJPS, and JOP from 2010 to 2022 and identify studies that use an IV strategy as one of the main identification strategies, including articles that use binary or continuous treatments and that use a single or multiple instruments. We use the following criteria: (1) the discussion of the IV result needs to appear in the main text and support a main argument in the paper; (2) we consider linear models only; in other words, articles that use discrete outcome models are excluded from our sample; \footnote{We expect the issue identified in this paper to be present, if not more pronounced, with nonlinear IVs. With nonlinear IVs, weak instruments correspond to weak identification in GMM estimations for some or all unknown parameters. Consequently, weak identification results in non-normal distributions even in large samples, rendering conventional IV or GMM inferences unreliable stock2002survey.} (3) we exclude articles that include multiple endogenous variables in a single specification (multiple endogenous variables in separate specifications are included); (4) we exclude articles that use IV or GMM estimators in a dynamic panel setting because the validity of the instruments (for example, $y_{t-2}$ affects $y_{t-1}$ but not $y_{t}$) is often not grounded in theories or substantive knowledge; these applications are subject to a separate set of empirical issues, and their poor performance has been discussed in the literature bun2010weak. These criteria result in 30 articles in the APSR, 33 articles in the AJPS, and 51 articles in the JOP. We then strive to find replication materials for these articles from public data-sharing platforms, such as the Harvard Dataverse, and the authors' websites. We are able to locate complete replication materials for 76 (62%) articles. However, code completeness and documentation quality vary widely. Since 2016-2017, data availability has significantly improved, thanks to new editorial policies that require authors to make replication materials publicly accessible key2016we. Starting in mid-2016 for AJPS and early-2021 for JOP, both journals introduced a policy requiring third-party verification of full replicability as a prerequisite for publication, although not all data are made public. We view these measures as significant advancements.
Using data and code from the replication materials, we set out to replicate the main IV results in these 76 articles with complete data. Our replicability criterion is simple: As long as we can exactly replicate one 2SLS point estimate that appears in the paper, we deem the paper replicable. We do not aim at exactly replicating SEs, $z$-scores, or level of statistical significance for the 2SLS estimates because they involve the choice of the inferential method. After much effort and hundreds of hours of work, we are able to replicate the main results of 67 articles.\footnote{For three articles, we are able to produce the 2SLS estimates with perfectly executable code; however, our replicated estimates are inconsistent with what was reported in the original studies. We suspect the inconsistencies are caused by data rescaling or misreporting; hence, we keep them in the sample.} The low replication rate is consistent with what is reported in Hainmueller2019-wx and chiu2023what. The main reasons for failures of replication are incomplete data (38 articles), incomplete code or poor documentation (4 articles), and replication errors (5 articles). Table (ref) presents summary statistics on data availability and replicability of IV articles for each of the three journals. The rest of this paper focuses on results based on these 67 replicable articles (and 70 IV designs).
\@startsection{paragraph}{4} \z@\z@{-\fontdimen2\font} *{Types of instruments.} Inspired by sovey2011instrumental, in Table (ref), we summarize the types of IVs in the replicable designs, although our categories differ from theirs to reflect changes in the types of instruments used in the discipline. These categories are ordered based on the strength of the design, in our view, for an IV study.
The first category is randomized experiments. These articles employ randomization, designed and conducted by researchers or a third party, and use 2SLS estimation to tackle non-compliance. With random assignment, our confidence in the exogeneity assumption increases because unconfoundedness is guaranteed by design and the direct effect of the instrument on the outcome is easier to rule out than without random assignment. For instance, alt2016 use assignment to an information treatment as an instrument for economic beliefs to understand the relationship between economic expectations and vote choice. Compared to IV articles published before 2010, the proportion of articles using experiment-generated IVs has increased significantly (from 2.9% to 17.1%) due to the growing popularity of experiments.
Another category consists of instruments derived from explicit rules on observed covariates, creating quasi-random variations in the treatment. sovey2011instrumental refer to this category as “Natural Experiment.” We avoid this terminology because it is widely misused and limit this category to two circumstances: fuzzy RD designs and variation in exposure to policies due to time of birth or eligibility. For example, kim2019 leverages a reform in Sweden that requires municipalities above a population threshold to adopt direct democratic institutions. dinas2014 uses eligibility to vote based on age at the time of an election as an instrument for whether respondents did vote. While rule-based IVs offer a pathway to credible causal inference, recent studies have raised concerns about their implementation, highlighting issues of insufficient power in many RD designs stommes2023reliability.
The next category is “Theory,” where the authors justify unconfoundedness and the exclusion restriction using social science theories or substantive knowledge. Over a decade after sovey2011instrumental's survey, it remains the most prevalent category among IV studies in political science, at around 60%. We divide theory-based IVs into four subcategories: geography/climate/weather, treatment diffusion, history, and others. First, Many studies in the theory category justify the choices of their instruments based on geography, climate, or weather conditions. For example, zhu2017 uses weighted geographic closeness as an instrument for the activities of multinational corporations; hager2019 use mean elevation and distance to rivers to instrument equitable inheritance customs; henderson2016mediating use rainfall around Election Day as an instrument for Democratic vote margins. Relatedly, several studies base their choices on regional diffusion of treatment. For example, dube2015 use US military aid to countries outside Latin America as an instrument for US military aid to Colombia. dorsch_maarek2019 use the regional share of democracies as an instrument for democratization in a country-year panel. \footnote{While authors often argue that weather or geography is quasi-randomly imposed, it is typically harder to claim they only affect the outcome through the treatment variable. For example, mellon2020rain contends that, while instruments like rain may be quasi-random, researchers have pinpointed several mechanisms through which it influences key political outcomes. betz2018use argue that spatial instruments are rarely valid because of cross-sectional interdependence and simultaneity. Inference also presents challenges.} Third, historical instruments derive from past differences between units unrelated to current treatment levels. For example, vernby2013 uses historical immigration levels as an instrument for the current number of non-citizen residents. Finally, several articles rely on a unique instrument based on theories that we could not place in a category. dower_etal2018 use religious polarization as an instrument for the frequency of unrest and argue that religious polarization could only impact collective action through its impact on representation in local institutions.
We wish to clarify that our reservations regarding instruments in this category are not primarily about theories themselves. As a design-based approach, the IV strategy requires specific and precise theories about the assignment process of the instruments and the exclusion restriction. We remain skeptical because many “theory”-driven instruments, in our view, do not genuinely uphold these assumptions, often appearing to be developed in an ad hoc or post hoc manner.
The last category of instruments are based on econometric assumptions. This category includes what sovey2011instrumental call “Lags.” These are econometric transformations of variables argued to constitute instruments. For example, lorentzen_etal2014 use a measure of the independent variable from eight years earlier to mitigate endogeneity concerns. Another example is shift-share “Bartik" instruments based. For example, baccini2021 use the interaction between job shares in specific industries and national employment changes to study the effect of manufacturing layoffs on voting. The number of articles relying on econometric techniques, including flawed empirical tests (such as regressing $y$ on $d$ and $z$ and checking if the coefficient of $z$ is significant), has decreased.
\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }{Replication Procedure and Results}
In this section, we describe our replication procedure and report the main findings.
\@startsection{paragraph}{4} \z@\z@{-\fontdimen2\font} *{Procedure.} For each paper, we select the main IV specification that plays a central role in supporting a main claim in the paper; it is either referred to as the baseline specification or appears in one of the main tables or figures. Focusing on this specification, our replication procedure involves the following steps. First, we compute the first-stage partial $F$-statistic based on (1) classic analytic SEs, (2) Huber White heteroskedastic-robust SEs, (3) cluster-robust SEs (if applicable and based on the original specifications), and (4) bootstrapped SEs. \footnote{They are calculated by $F_{boot} = \hat\tau_{2SLS}'{\hat{\mathbb{V}}_{boot}(\hat\tau_{2SLS})}^{-1}\hat\tau_{2SLS}/p_{z}$, where $p_{z}$ is the number of IVs and $\hat{\mathbb{V}}_{boot}(\hat\tau_{2SLS})$ is the estimated variance-covariance matrix based on a nonparametric bootstrap procedure, in which we repeatedly sample the rows of the data matrix with replacement. If the data have a clustered structure, we use cluster-bootstrapping instead Colin_Cameron2015-wp,Esarey2019-qt. We include $F_{boot}$ as a reference to the classic $F$ and effective $F$. In Section A2 of the SM, we compare the five types of $F$-statistics and show that the effective $F$ and $F$ based on bootstrapping are usually more conservative (smaller) than other $F$-statistics.} We also calculate $F_{\texttt{Eff}}$.
We then replicate the original IV result using the 2SLS estimator and apply four different inferential procedures. First, we make inferences based on analytic SEs, including robust SEs or cluster-robust SEs (if applicable). Additionally, we use two nonparametric bootstrap procedures, as described in Section 2, bootstrap-c and bootstrap-t. For specifications with only a single instrument, we also employ the $tF$ procedure proposed by Lee2020-mi, using the 2SLS $t$-statistic and first-stage $F$-statistic based on analytic SEs accounting for the originally specified clustering structure. Finally, we conduct an AR procedure and record the $p$-values and CIs.
We record the point estimates, SEs (if applicable), 95% CIs, and $p$-values for each procedure (the point estimates fully replicate the reported estimates in the original articles and are the same across all procedures). In addition, we estimate a naïve OLS model by regressing the outcome variable on the treatment and covariates, leaving out the instrument. We calculate the ratio between the magnitudes of the 2SLS and OLS estimates, as well as the ratio of their analytic SEs. We also record other useful information, such as the number of observations, the number of clusters, the types of instruments, the methods used to calculate SEs or CIs, and the rationale for each paper's IV strategy. Our replication yields the following three main findings.
\@startsection{paragraph}{4} \z@\z@{-\fontdimen2\font} *{Finding 1. The first-stage partial $F$-statistic.} Our first finding regards the strengths of the instruments. To our surprise, among the 70 IV designs, 12 (17%) do not report this crucial statistic despite its key role in justifying the validity of an IV design. Among the remaining 58 studies that report $F$-statistic, 9 (16%) use classic analytic SEs, thus not adjusting for potential heteroskedasticity or clustering structure. In Figure (ref), we plot the replicated first-stage partial $F$-statistic based on the authors' original model specifications and choices of variance estimators on the x-axis against (a) effective $F$-statistic or (b) bootstrapped $F$-statistic on the y-axis, both on a logarithmic scale.\footnote{We use the replicated $F$-statistics instead of the reported ones because some authors either do not report or misreport their $F$-statistics (see SM for a comparison between the reported and replicated $F$-statistics).}
In the original studies, the authors used various SE estimators, such as classic SEs, robust SEs, or cluster-robust SEs. As a result, the effective $F$ may be larger or smaller than the original ones. However, a notable feature of Figure (ref) is that when a clustering structure exists, the original $F$-statistic tends to be larger than the effective $F$ or bootstrapped $F$. When using the effective $F$ as the benchmark, 8 studies (11%) have $F_{\texttt{Eff}}<10$. This number increases to 12 (17%) when the bootstrapped $F$-statistic is used. The median first-stage $F_{\texttt{Eff}}$ statistic is higher in experimental studies compared to non-experimental ones (67.7 versus 53.5). It is well known that failing to cluster the SEs at appropriate levels or using the analytic cluster-robust SE with too few clusters can lead to an overstatement of statistical significance cameron2008bootstrap. However, this problem has received less attention when evaluating IV strength using the $F$-statistic. \footnote{abadie2020sampling and Abadie2022 delineate the differences between a traditional sampling-based view, where clustering arises from a two-stage sampling process (sampling clusters, then units within them), and a design-based view, where clustering stems from the clustered nature of treatment assignment. The key takeaway from both papers is the importance of clustering at the unit of randomization. They argue that finite-population standard errors, rooted in the design-based perspective, can be tighter than conventional cluster-robust errors. Given that the exact design is often unknown in many political science observational studies, clustering where the instrument is likely assigned offers a more reliable approach for valid inference. In the replication exercise, however, we cluster SEs according to the levels specified by the original authors.}
\@startsection{paragraph}{4} \z@\z@{-\fontdimen2\font} *{Finding 2. Inference.} Typically, 2SLS estimates have higher uncertainties than OLS estimates. Figure (ref) reveals that the 2SLS estimates in the replication sample are in general much less precise than their OLS counterparts, with the median ratio of the analytic SEs equal to 3.8. This ratio decreases as the strength of the instrument, measured by the estimated correlation coefficient between the treatment and predicted treatment $|\hat\rho(d, \hat{d})|$, increases. This is not surprising because $\hat\rho(d, \hat{d})^2 = R_{dz}^2$, the first-stage partial $R$-squared. However, one important implication of large differences in SEs is that to achieve comparable levels of statistical significance, 2SLS estimates often need to be at least three times larger than OLS estimates---not to mention that $t$-testing based on analytical SEs for 2SLS coefficients is often overly optimistic. This difference in precision sets the stage for potential publication bias and $p$-hacking.
Next, we compare the reported and replicated $p$-values for the null hypothesis of no effect. For studies that do not report a $p$-value, we calculate it based on a standard normal distribution using the reported point estimates and SEs. The replicated $p$-values are based on (1) bootstrap-c, (2) bootstrap-t, and (3) the AR procedure. Since we can exactly replicate the point estimates for the articles in the replication sample, the differences in $p$-values are the result of the inferential methods used. Figure (ref)(a)-(c) plot reported and replicated $p$-values, from which we observed two patterns. First, most of the reported $p$-values are smaller than 0.05 or 0.10, the conventional thresholds for statistical significance. Second, consistent with Young2022's finding, our replicated $p$-values based on the bootstrap methods or AR procedure are usually bigger than the reported $p$-value (exceptions are mostly caused by rounding errors), which are primarily based on $t$ statistics calculated using analytic SEs. Using the AR test, we cannot reject the null hypothesis of no effect at the 5% level in 12 studies (17%), compared with 7 (10%) in the original studies. The number increases to 13 (19%) and 19 (27%) when we use $p$-values from the bootstrap-t and -c methods. Note that very few articles we review utilize inferential procedures specifically designed for weak instruments, such as the AR test (2 articles), the conditional likelihood-ratio test Moreira2003-oj (1 paper), and confident sets Mikusheva2006-lk (none).
We also apply the $tF$ procedure to 54 studies that use single IVs using $F_{\texttt{Eff}}$ statistics and $t$ statistics based on robust or cluster-robust SEs. Figures (ref)(d) shows that 19 studies (35%) are not statistically significant at the 5% level, and 7 studies (13%) deemed statistically significant when using the conventional fixed critical values for the $t$-test become statistically insignificant using the $tF$ procedure, indicating that overly optimistic critical values due to weak instruments also contribute to overestimation of statistical power, but not as the primary factor. These results suggest that both weak instruments and non-i.i.d. errors have contributed to overstatements of power in IV studies in political science.
\FloatBarrier
\@startsection{paragraph}{4} \z@\z@{-\fontdimen2\font} *{Finding 3. 2SLS-OLS discrepancy.} Finally, we investigate the relationship between the 2SLS estimates and naïve OLS estimates. In Figure (ref)(a), we plot the 2SLS coefficients against the OLS coefficients, both normalized using reported OLS SEs. The shaded area indicates the range beyond which the OLS estimates are statistically significant at the 5% level. It shows that for most studies in our sample, the 2SLS estimates and OLS estimates share the same direction and that the magnitudes of the former are often much larger than those of the latter. Figure (ref)(b) plots the distribution of the ratio between the 2SLS and OLS estimates (in absolute terms). The mean and median of the absolute ratios are 12.4 and 3.4, respectively. In fact, in all but two designs (97%), the 2SLS estimates are bigger than the OLS estimates, consistent with jiang2017have's finding based on finance research. While it is theoretically possible for most OLS estimates in our sample to be biased towards zero, only 21% of the studies have researchers expressing their belief in downward biases of the OLS estimates. Meanwhile, 40% of the studies consider the OLS results to be their main findings. The fact that researchers use IV designs as robustness checks for OLS estimates due to concerns of upward biases is apparently at odds with the significantly larger magnitudes of the 2SLS estimates.
\FloatBarrier
In Figure (ref)(c), we further explore whether the 2SLS-OLS discrepancy is related to IV strength, measured by $|\hat\rho(d, \hat{d})|$. We find a strong negative correlation between $|\hat{\tau}_{2SLS} / \hat{\tau}_{OLS}|$ and $|\hat\rho(d,\hat{d})|$ among studies using non-experimental instruments (grey dots). The adjusted $R^2$ is $0.264$, with $p = 0.000$. However, the relationship is much weaker among studies using experiment-generated instruments (red dots). The adjusted $R^2$ is $-0.014$ with $p = 0.378$. At first glance, this result may seem mechanical: as the correlation between $d$ and $\hat{d}$ increases, the 2SLS estimates naturally converge to the OLS estimates. However, the properties of the 2SLS estimator under the identifying assumptions do not predict the negative relationship (we confirm it in simulations in the SM), and such a relationship is not found in experimental studies. In Figure (ref)(d), we limit our focus to the subsample in which the OLS estimates are statistically significant at the 5% level and researchers accept them as (part of) the main findings, and the strong negative correlation remains.
Several factors may be contributing to this observed pattern, including (1) failure of the exogeneity assumption, (2) publication bias, (3) heterogeneous treatment effects, and (4) measurement error in $d$. As noted earlier, biases originating from endogenous IVs or exclusion restriction failures can be magnified by weak instruments, i.e., $\frac{|\text{Bias}_{IV}|}{|\text{Bias}_{OLS}|} = \left|\frac{\text{Cov}(z, \varepsilon)\mathbb{V}\left[d\right]}{ \text{Cov}(d, \varepsilon)\text{Cov}(z, d)}\right| = \frac{|\rho(z, \varepsilon)|}{|\rho(d, \varepsilon)|\cdot |\rho(d, \hat{d})|} \gg~1$. In addressing large IV-OLS estimate ratios, hahn2005estimation suggest two explanations: it could stem from a bias in OLS or from a bias in IV due to violations of the exogeneity assumption. Our empirical results, with particularly dubious IV to OLS estimate ratios in non-experimental studies, seem to align with the latter explanation.
Publication bias may have also played a significant role. As shown in Figure (ref), the variance of IV estimates increase as $|\hat\rho(d,\hat{d})|$ diminishes. If researchers selectively report only statistically significant results, or if journals have a tendency to publish such findings, it is not surprising that the discrepancies between IV and OLS estimates widen as the strength of the first stage declines, as shown in Figure (ref)(c)-(d). This is because 2SLS estimates often need to be substantially larger than OLS estimates to achieve statistical significance. This phenomenon is known as Type-M bias and has been discussed in psychology and sociology literature gelman2014beyond, FeltonStewart2022. Invalid instruments exacerbate this issue by providing ample opportunities for generating such large estimates.
Moreover, 30% of the replicated studies in our sample mention heterogeneous treatment effects as a possible explanation for this discrepancy. OLS and 2SLS place different weights on covariate strata in the sample, and therefore if compliers, those whose treatment status is affected by the instrument, are more responsive to the treatment than the rest of the units in the sample, we might see diverging OLS and 2SLS estimates. Under the assumption that the exclusion restriction holds, this gap can be decomposed into covariate weight difference, treatment-level weight difference, and endogeneity bias components using the procedure developed in Ishimaru2021-ik. In the SM, we investigate this possibility and find that it is highly unlikely that heterogeneous treatment effects alone can explain the difference in magnitudes between 2SLS and OLS estimates we observe in the replication data, i.e., the variance in treatment effects needed for this gap is implausibly large.
Finally, IV designs can correct for downward biases due to measurement errors in $d$, resulting in $|\hat{\tau}_{2SLS} / \hat{\tau}_{OLS}| > 1$. If the measurement error is large, this can weaken the relationship between $d$ and $\hat{d}$, producing a negative correlation. We find it an unlikely explanation because only four articles (6%) attribute their use of IV to measurement errors, and the negative correlation is even stronger when we focus solely on studies where OLS estimates are statistically significant and regarded as the main findings.
In Table (ref), we present the main findings from our replication exercise. Observational studies, compared to experimental counterparts, generally have weaker first stages, often display larger increases in $p$-values when more robust inferential methods are used, and demonstrate bigger discrepancies between the 2SLS and OLS estimates. Based on these findings, we contend that a significant proportion of IV results based on observational data in political science either lack credibility or yield estimates that are too imprecise to offer insights beyond those provided by OLS regressions.
\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }{Recommendations}
IV designs in experimental and observational studies differ fundamentally. In randomized experiments, the instruments' unconfoundedness is ensured by design, and researchers can address possible exclusion restriction violations at the design stage, e.g., by testing potential design effects through randomization Gerber2012-fr. Practices like power analysis, placebo tests, and preregistration also help reduce the improper use of IVs. In contrast, observational IV designs based on “natural experiments” require detailed knowledge of the assignment mechanism, making them more complex and prone to issues sekhon2012natural.
Our findings suggest that using an IV strategy in observational settings is extremely challenging due to several reasons. First, truly random and strong instruments are rare and difficult to find. This is mainly because neither unconfoundedness nor the exclusion restriction is guaranteed by design, placing a greater burden of proof on researchers for the exogeneity assumption. Moreover, conducting placebo tests like the ZFS test for the exclusion restriction after data collection is not always feasible. Finally, increasing the sample size to achieve sufficient statistical power is often impractical. To prevent misuse of IVs in observational studies, we provide a checklist for researchers to consider when applying or contemplating an IV strategy with one endogenous treatment variable:
\FloatBarrier
We provide an R package, \href{https://github.com/apoorvalal/ivDiag}{ivDiag}, to implement our recommended procedures. Stata tutorials for carrying out these procedures are also available on the corresponding author's website. Our aim is to address concerns regarding IVs in social science research and improve the quality of estimation and inference, especially for non-experimental IV designs.
\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }*{Acknowledgements} We thank Te Bao, Thomas Cao, Albert Chiu, Daniel Chen, Gary Cox, Charles Crabtree, Ted Enamorado, Hanming Fang, Avi Feller, Don Green, Justin Grimmer, Anna Grzymala-Busse, Jens Hainmueller, Guido Imbens, David Laitin, Adeline Lo, Justin McCrary, Jacob Montgomery, Doug Rivers, Henrik Sigstad, Brandon Stewart, Arthur Yu, and Xiang Zhou, as well as seminar participants at Stanford University, Washington University in St. Louis, APSA 2021, and Polmeth 2021, for their extremely helpful comments. We are also deeply grateful to four anonymous reviewers for PA and Editor Jeff Gill, as well as two anonymous reviewers for APSR---their invaluable suggestions have improved this paper significantly.
\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }*{Data Availability Agreement} Replication data and code for this article have been published at Harvard Dataverse at \url{https://doi.org/10.7910/DVN/MM5THZ}.
\@startsection{section}{1} \z@{.7\linespacing\@plus\linespacing}{.5\linespacing} {\normalfont }*{Supplementary Material} For supplementary material accompanying this paper, please visit \url{https://doi.org/10.xxxx/ pan.20xx.xx}.
\singlespacing \printbibliography