Clara Bicalho, Adam Bouyamourn, Thad Dunning
arXiv 21 May 2022 · Statistics — Methodology · 2 citations (OpenAlex)
arXiv:2205.10478 · PDF · DOI · OpenAlex · Extracted main text
Scholars frequently use covariate balance tests to test the validity of natural experiments and related designs. Unfortunately, when measured covariates are unrelated to potential outcomes, balance is uninformative about key identification conditions. We show that balance tests can then lead to erroneous conclusions. To build stronger tests, researchers should identify covariates that are jointly predictive of potential outcomes; formally measure and report covariate prognosis; and prioritize the most individually informative variables in tests. Building on prior research on “prognostic scores," we develop bootstrap balance tests that upweight covariates associated with the outcome. We adapt this approach for regression-discontinuity designs and use simulations to compare weighting methods based on linear regression and more flexible methods, including machine learning. The results show how prognosis weighting can avoid both false negatives and false positives. To illustrate key points, we study empirical examples from a sample of published studies, including an important debate over close elections.
appendix boundary found by appendix_titled_section at “Technical appendix” · 77% of the source is main text. Read the extracted text to check this.
The works this paper leans on most, across its whole bibliography — not restricted to papers in our corpus. Ranked by composite intensity, which combines how often a work is mentioned, how many sections mention it, and how much of that falls in the main text rather than the appendix.
| Reference | Intensity | Mentions | Sections | Main text | |
|---|---|---|---|---|---|
| 1 | Caughey, D., Dafoe, A., and Seawright, J (2017) Nonparametric combination (npc): A framework for testing elaborate theories | 0.941 | 6 | 3 | 83% |
| 2 | Stuart, E., Lee, B., and Leacy, F (2013) Prognostic score-based balance measures can be a useful diagnostic for propensity score methods in comparative effectiveness res… | 0.941 | 6 | 3 | 83% |
| 3 | Boas, T. C. and Hidalgo, F. D (2011) Controlling the airwaves: Incumbency advantage and community radio in brazil | 0.928 | 5 | 3 | 80% |
| 4 | Eggers, A. C., Fowler, A., Hainmueller, J., Hall, A. B., and Snyder,… (2015) On the validity of the regression discontinuity design for estimating electoral effects: New evidence from over 40,000 close races | 0.874 | 8 | 2 | 100% |
| 5 | De la Cuesta, B. and Imai, K (2016) Misunderstandings about the regression discontinuity design in the study of close elections | 0.855 | 8 | 4 | 62% |
| 6 | Blattman, C (2009) From violence to voting: War and political participation in uganda | 0.843 | 4 | 3 | 75% |
| 7 | Hansen, B. B (2008) The prognostic analogue of the propensity score | 0.843 | 5 | 3 | 60% |
| 8 | Caughey, D. and Sekhon, J. S (2011) Elections and the regression discontinuity design: Lessons from close u.s. house races, 1942-2008 | 0.811 | 4 | 2 | 100% |
| 9 | Dunning, T (2012) Natural Experiments in the Social Sciences: A Design-Based Approach self | 0.737 | 4 | 3 | 50% |
| 10 | Fisher, R. A (1935) The design of experiments | 0.737 | 3 | 3 | 67% |
Showing the top 10 of 70 scored citations.
arXiv econ.EM papers that cite this one, ranked by how heavily they lean on it.
| Citing paper | Intensity | Mentions | Sections | |
|---|---|---|---|---|
| 1 | Integrating Diagnostic Checks into Estimation | 0.511 | 2 | 1 |